DOCUMENT RESUME 

ED 381 631 CE 068 722 



AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENCY 
PUB DATE 
NOTE 
PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Reder, Stephen; Wikelund, Karen Reed 

Steps to Success: Literacy Development in a 

Welfare-to-Work Program. Final Report. 

Northwest Regional Educational Lab., Portland, 

Oreg. 

National Inst, for Literacy, Washington, DC. 

Nov 94 

77p. 

Reports - Research/Technical (143) — 
Tests/Evaluation Instruments (160) 

MF01/PC04 Plus Postage. 

Adult Basic Education; Basic Skills; Career Planning; 
'''Educational Benefits; Educational Research; 
Education Work Relationship; Followup Studies; 
'^Literacy Education; Low Income Groups; Program 
Effectiveness; ''Welfare Recipients 
Life Planning; Welfare to Work Programs 



ABSTRACT 

A research project investigated long-term literacy 
development and its economic impact among clients of a 
welfare-to-work program. A follow-up study was conducted with a 
sample of 920 former clients of Steps to Success's educational track, 
which consisted of a career and life planning course followed by 
adult literacy education. From the sample, 229 agreed to participate 
and 163 actually completed a basic skills test and a questionnaire 
about their post-program activities. These participants had started 
the program and taken an initial basic skills test 1-3 years earlier. 
Data came from several sources: standardized tests of literacy 
skills; questionnaire data about post-program education, training, 
and employment activities and client perceptions of program impact 
and quality; "hard" data about clients' subsequent employment, 
earnings, and welfare dependence; and qualitative data from program 
staff and a cohort of clients followed closely over several years. 
Results indicated that clients raised their literacy scores 
substantially. These assessed skill gains were larger than those 
typically reported by adult education programs and represented 
sustained learning gains. Strong indications were that clients' 
assessed literacy gains were significantly associated with reductions 
in their future dependence on public assistance. Data analysis 
identified three principal factors underlying clients' outcomes and 
perceptions: employment, personal growth, and educational 
development. (Appendixes contain a list of 47 references and the 
questionnaire.) (YLB) 
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ABSTRACT 



This research project investigated long-term literacy development and its economic impact 
among clients of a welfare-to-work program A follow-up study was conducted with a 
sample of former clients of the program's educational track, which consisted of a Career 
and Life Planning course followed by adult literacy education. The study participants took 
a basic skills test and completed a questionnaire about their post-program activities. 
These participants had started the program and had taken an initial basic skills test one to 
three years before the study's testing and questionnaire activities. Data analyzed in the 
study came from a rich variety of sources: standardized tests of literacy skills; 
questionnaire data out post-program education, training and employment activities; 
questionnaire data about client perceptions of program impact and quality; "hard" data 
about clients' subsequent employment, earnings and welfare dependence; and rich 
qualitative data from program staff and a cohort of clients followed closely over several 
years. 

Results of the study indicate that clients in the educational track of the welfare-to-work 
program raised their literacy skills substantially. These assessed skill gains, measured as 
differences between scores on pre-tests administered at program entry and post-tests 
administered by the study some one to three years later, not only are larger than those 
typically reported by adult education programs, but they represent sustained learning 
gains, measures of long-term literacy growth that persist years after program exit. Nearly 
all clients, regardless of entering skill level, experienced this literacy growth. These 
findings add significantly to the hard evidence that such broad and sustained literacy 
development occurs among adult education students. Furthermore, there are strong 
indications that clients' assessed literacy gains are significantly associated with reductions 
in their future dependence on public assistance. Differences between these findings and 
those of California's Greater Avenues to Independence (GAIN) evaluation are considered 
carefully. 

Analysis of the various data collected during the study identifies three principal factors 
underlying clients' experiences of the welfare-to-work program and its impact on their 
long-term literacy development and economic outcomes. These three factors are termed 
Employment, Personal Growth and Educational Development. Clients' goals and 
outcomes appear to encompass these three factors with varying emphases. Interestingly, 
perceived improvements in basic skills are associated with Personal Growth but not with 
Educational Development (which encompasses assessed gains in literacy skills). Much of 
the qualitative information from program client and staff interviews and focus groups 
corroborates this finding insofar as clients tend to view improvements in their basic skills 
as an inseparable component of their overall personal growth rather than as the 
development of a free-standing set of skills. This may well be one reason why 
standardized basic skills tests are widely seen as not being aligned with what the clients 
(and perhaps most other adult students) learn in literacy programs. The finding that 
student perceptions of literacy development do not align with assessed skill gains may 



have major implications for the design of adult literacy programs, whether they serve 
welfare clients or other adult learners. 

These findings have important theoretical and practical implications. The study provides 
important new evidence of broad, sustained literacy development in adult students. There 
has been little previous evidence of such learning gains that has not been contaminated by 
artifacts of repeated testing and selective learner attrition. Such findings should help 
bolster theories of long-term literacy growth. The strong (but not necessarily decisive) 
indications in the study that such learning gains among welfare clientele are linked to 
reduced future dependency rates has important policy implications for upcoming welfare 
reform. The organization of client experiences into the three dimensions of Employment, 
Personal Growth and Educational Development could serve as a useful framework for the 
design of more effective adult education and welfare-to-work programs. 

The Career and Life Planning course, whose impact unfortunately could not be statistically 
isolated from that of the. literacy education which followed it, appears to be a broadly 
applicable program element through which literacy education can be integrated with 
personal growth and employment-oriented program components. Specific 
recommendations include: (1) blend educational, personal growth and work-transition 
program components in a carefully designed manner; (2) integrate literacy instruction with 
other services supporting personal growth; and (3) develop effective ways of providing 
continuing support services after clients secure employment and begin working. 
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DEDICATION AND PREFACE 



This paper is dedicated to the memory, life and work of Karen Reed Wikelund. Ms. 
Wikelund, a close friend and colleague of mine during the past 16 years, was a gifted 
researcher and powerful community activist committed to improving the standard of living 
and quality of life for those living in poverty, especially individuals limited by their 
education, language and literacy backgrounds. Karen applied much of her abundant 
talents, energy and enthusiasm to helping families break out of and escape from the 
cyclical constraints of poverty, unemployment, undereducation and low self-esteem 

Karen's ongoing program of research on the impact of life planning, motivation and basic 
education in helping single mothers break out of the welfare system (Wikelund, 1993) was 
the foundation of the present project. Karen was the lead researcher for this project, 
which was designed to build on her ongoing research about the life experiences and 
educational development of a cohort of welfare mothers she had been closely studying for 
several years, beginning with their entrance into the career and life planning component of 
a model welfare reform program Planning to rely on her rich qualitative understanding of 
the cohort as an interpretive guide, our project collected a quasi-exp erimental follow-up 
"snapshot" of a larger and more representative sample of individuals who had previously 
participated in the program By combining Wikehind's rich ethnographic data and 
understanding with the follow-up survey "snapshot", the project aimed to collect evidence 
about the impact of combining career and life planning with adult education services in 
forming a more responsive and effective "welfare reform" system 

Tragically, Karen developed lymphoma midway through the project. Her heroic battle 
against the deadly disease did not lessen her resolve or commitment to her work, but 
progressively diminished her capacity to conduct the field research with the cohort of 
women with whom she had been working so closely. Despite the hopeful signs of a brief 
remission last Spring, Karen passed away on September 24, 1994. 

Along with her family, Karen's many friends, colleagues and "field acquaintances" are 
deeply saddened by her untimely death. Her professional and community contributions 
will be acknowledged and remembered in many other places. She would want us all to 
continue the struggle to make literacy an empowering, progressive force within individual, 
family and community life. I do believe her integrity, commitment and insights will 
continue to help us move forward on that path in years to come. But we have lost a most 
valuable and irreplaceable ally ux this endeavor. 
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I am saddened and humbled by the task of trying to bring some closure here to what must 
remain an essentially unfinished project. Karen's knowledge, insight and voice would have 
added enormously to this report. Her finely honed ethnographic outlook on and in-depth 
knowledge of the experiences of women who passed through the Steps to Success 
program undoubtedly would greatly enrich our understanding and interpretation of the 
quantitative survey data that I have struggled to make sense of in Karen's absence. ■ 
Unfortunately, the timing of Karen's illness prevented her from participating in the 
preparation of this report As if that gap were not enough, the moving and insightful 
voices of the women she was working with are also very quiet here. Karen and I hoped 
this project might amplify their voices to the point that they might be heard by those 
shaping future welfare reform 

If the research, reported here without Karen's insight and voice or those of her "friends in 
the field", nevertheless assists this effort in some small way, I shall be very gratified. As 
limited as this report must be without her active involvement, I am delighted to dedicate it 
very humbly to Karen Wikehind's memory and progressive vision. 



-Steve Reder 
Portland, Oregon 
November 4, 1994 
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INTRODUCTION 



Universal adult literacy and lifelong learning have recently been enacted as one of our 
National Education Goals. The standard envisioned for adult literacy is no longer the 
mere ability to read and write, but the possession of the literacy skills and knowledge 
needed to compete successfully in the workforce and global economy and to participate 
actively as a citizen in our democracy. The millions of American adults without adequate 
literacy skills and knowledge are experiencing increasingly limited opportunities for 
employment, earnings and economic self-sufficiency for themselves and their families. The 
society as a whole also has less opportunity to prosper economically and socially when too 
many of its members do not develop the literacy skills and knowledge needed to function 
as productive workers and citizens. 

This report describes the results and implications of efforts designed to improve the 
literacy skills and economic self-sufficiency of a group of adults which historically has 
lacked such skills: those depending on public assistance to support their families. The 
report begins with a brief overview of the history and scope of the public welfare system 
and the crucial role which literacy appears to play in welfare dependence. That history 
leads into a brief discussion of differing views of the role for literacy training within 
emerging approaches to welfare reform. In considering these options for welfare reform, 
a number of important research questions are identified regarding the impact of adult 
education on literacy proficiencies and economic outcomes such as employment, earnings 
and welfare dependence. A review of past research indicates that more needs to be known 
about these relationships to serve as a foundation for developing sound welfare reform 
policies and programs. 

The report then goes on to describe a study of these relationships among clients of a 
model welfare-to-work program, Steps to Success. Results of this study provide some 
important new information about the efficacy of efforts to increase welfare clients' literacy 
skills and the impact of increasing skills on future economic self-sufficiency. The 
implications of the findings are discussed in the final section of the report. 

Literacy and Welfare: A Brief Overview 

The growing scope of the welfare system. The major cash assistance program in the 
nation's welfare system is Aid to Families with Dependent Children (AFDC). In 1992 
about 4.9 million families received AFDC, encompassing about 14.0 million recipients, 
more than two-thirds of whom (9.5 million) were children. The average monthly payment 
per family was $381. Total expenditures for these AFDC benefits exceeded $22. 1 billion 
in 1992, of which the federal share was about $13 million and the states' share was about 
$9 billion (U.S. Bureau of the Census, 1994) When all federal costs for supporting 
program clientele are included (AFDC, medicaid, food stamps, and administrative costs), 
welfare expenditures are approximately 1% of the federal budget (Youth Policy, 1993). 




Established in 1935 to help meet the needs of poor children in single-parent households, 
the AFDC program was originally intended to assist widows in poverty to stay at home to 
care for their children. Program size and costs grew dramatically in the ensuing decades 
as single mothers on assistance increasingly were divorced, separated or never married 
rather than widowed. AFDC caseloads have risen steadily in recent decades, up nearly 
83% between 1970 and 1992, representing a constant-dollar increase of over 51% 
(Martinson, 1994). Despite the steady growth of the program, the majority of clients 
receive cash assistance for relatively short periods of time. A small fraction of AFDC 
clientele are long-term recipients, although they account for a significant percentage of the 
overall costs (Bane & Ellwood, 1983). 

Despite these increasing AFDC caseloads, the average cash benefit per family has 
markedly decreased in constant-dollars since the 1970s; having dropped some 25%. Not - 
surprisingly, these smaller real benefits are able to lift a correspondingly smaller 
percentage of client families above the poverty line, now estimated to be about 1 in 7 
compared to 1 in 5 two decades ago (Youth Policy, 1993). These trends towards 
increasing caseloads with decreasing real benefits have continued in recent years. In the 
approximate three years between July 1989 and October 1993, for example, the national 
AFDC caseload grew 34% from 3.75 million to 5.02 million families (Youth Policy, 
1993). 

The emerging role ofliteracy and adult education within the welfare system. It has 
long been clear that lack of educational credentials and basic skills pose serious barriers to 
poor families becoming economically self-sufficient. Since at least the 1960s, welfare 
programs have tried a variety of approaches to assist their clients to develop the basic 
skills and educational credentials that will enable them to participate in job training and 
secure employment that provides livable wages and benefits for their families. 

After 10-15 years of programmatic experimentation with a variety of such human capital 
approaches to increasing the employability of welfare clientele, Congress passed the 
Family Support Act of 1988 which attempted to codify what had been learned through this 
experimentation. The Job Opportunities and Basic Skills Training Program (JOBS), a 
central component of the Family Support Act, is based on three interrelated assumptions: 
(1) AFDC parents are responsible for supporting their families; (2) government is 
responsible to assist AFDC parents to develop the skills they need to support their 
families; and (3) strong literacy and basic skills are essential for AFDC parents to 
successfully make the transition into employment and economic self-sufficiency. 

Under the JOBS program, a total of approximately $1 billion in Federal funds is allocated 
per year to the states on a formula basis according to their number of AFDC recipients. 
To obtain their full share of federal funds for JOBS, states must meet certain requirements, 
which include providing matching state funds and serving a minimum percentage of high 
risk target groups (e.g., long-term AFDC recipients, custodial parents with little education 
or work experience, and families about to lose their AFDC eligibility). 
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Literacy education is the centerpiece of most states' JOBS programs. At any given time, 
more JOBS participants are enrolled in basic education than in any other JOBS activity 
(Chisman & Woodworm, 1992). According to federal regulations, JOBS programs must 
target raising participants basic skills to the equivalent of at least an 8.9 grade level. Most 
states, however, found the federal target skill level unrealistically low for securing jobs 
paying livable wages. Many opted for the more demanding target of obtaining a General 
Educational Development (GED) certificate (Chisman & Woodworth, 1992). 

Close looks at the ways in which states have implemented the JOBS program (e.g., 
Chisman & Wcodworth, 1992; Friedlander, Riccio & Freedman, 1993) have raised serious 
questions in some quarters about its effectiveness at moving AFDC clientele from welfare 
to work. Although there is widespread consensus about the desirability of the Clinton 
administration's announced intention of "ending welfare as we know it", there is little 
agreement yet about the detailed shape such welfare reform should take. Among other 
issues to be resolved is the relative intensity and duration of basic education as a 
component of a reformed welfare system. The administration favors a plan that involves 

"a two-year time limit on cash assistance and a greater emphasis on 'promoting self- 
sufficiency through access to education and training.' The goal is to assure that by the 
end of two years adult recipients of Aid to Families with Dependent Children (AFDC) 
have the skills, knowledge and resources necessary to achieve self-support and find 
work in the private sector. Those unable to find private sector jobs will be supported 
via community service jobs." (National Institute for Literacy, 1994) 

The Policy Context for the Research Project 

To assist policy makers to consider emerging policy alternatives carefully, it is 
essential that they are given accurate and useful information about the likely 
implications of including basic skills training in future welfare programs. Given the 
public cost and family support implications associated with the alternative proposals 
being considered, it is surprising that so little is presently known about literacy 
development and its economic consequences among welfare program clientele. There 
are several major questions about literacy and welfare that decision-makers need to 
know. First, what evidence is there that literacy (when education is held constant) is a 
factor in welfare dependence? Second, to what extent does participation in adult 
literacy education increase the literacy skill! of welfare clients? Third, what are the 
economic consequences of raising the literacy proficiencies of welfare clients (e.g., its 
impact on future employment, earnings, poverty and dependency rates)? Basting 
research on these points is briefly reviewed below. 

What evidence links education and literacy problems to welfare clients? Most of the 
evidence linking education and literacy to welfare dependence is cross-sectional in nature. 
That is, the evidence compares characteristics of those who do and do not receive public 
assistance such as AFDC. According to Martinson (1994), about 46% of the AFDC 



caseheads do not have a high school diploma or equivalent (compared to 25% of the 
general adult population no longer in school). Individuals without these secondary 
credentials are facing an increasingly difficult time finding and maintaining jobs that pay 
well enough to meet their families' basic needs. Many programs attempt to remedy their 
clients' lack of educational credentials by either assisting young clients to return to high 
school or by encouraging clients to prepare for, take and pass the GED tests (Bloom, 
Fellerath, Long & Wood, 1993; Pauly & Martinson, 1992). 

The lack of basic literacy skills is also thought to be significant problem for many AFDC 
caseheads and their families. Although some welfare analysts have attempted to make 
sweeping generalizations about the relationship between literacy and welfare dependence 
without careful analysis (e.g., Marsh, Pollan, McFadden & Price, 1990), Berlin and Sum 
(1988) have intensively examined the empirical data and have clearly shown that poor 
basic skills and lack of education are each strongly associated with employment, earnings, 
poverty and welfare dependence. Although some individual, programs and states are 
beginning to assess the basic skills of their clientele (e.g., Hughes, 1991; Employment 
Division, 1993), there has been relatively little systematic description of the literacy 
abilities of the nation's welfare clientele. 

The recently completed National Adult Literacy Survey (NALS) offers a more complete 
picture of the literacy proficiencies of the nation's welfare clientele. The NALS provides 
detailed information about a random sample of the U.S. adult population ages 16 and 
older. In addition to background information, the NALS included a functional literacy 
assessment which asked respondents to perform simulated literacy tasks such as filling in 
forms, extracting information from printed materials and graphical displays, and locating 
needed information in text, on maps, and so forth (Kirsch, Jungeblut, Jenkins & Kolstad, 
1993). Individuals'iiteracy proficiencies were estimated as scores on three 0-500 point 
scales: prose, document and quantitative literacy. 

Martinson (1994) and Rickard (1994) have used these NALS data to profile the distinctive 
literacy abilities of the nation's AFDC population. Martinson, for example, notes that the 
AFDC population scores about 30 points below the general adult population on each 
scale, a large difference corresponding to about 1/2 standard deviation 1 . Youth Policy 



'Both Rickard (1994) and Martinson (1994) present a NALS "AFDC" profile as though the respondents 
involved were AFDC caseheads. Most of these "AFDC" respondents, however, clearly are not AFDC 
caseheads. NALS respondents were asked on a yes-or-no basis whether "anyone in your household 
receives AFDC, public assistance or public welfare payments..." (Educational Testing Service, 1991). 
Receipt of SSI was asked about in the immediately preceding question, so that it presumably was not 
involved in answers to this question. The group of NALS respondents answering "yes" have a Census- 
weighted size of about 12 million, which corresponds closely with the total number of AFDC recipients at 
the time (of whom about 2/3 are known to be children under 16 and thus not in the NALS population). 
Thus only about 1/3 of this group of respondents could be AFDC caseheads. Furthermore, the 
demographic characteristics of the NALS "AFDC" group differ markedly from those reported elsewhere 
for AFDC caseheads (e.g., distributions of age, gender, education, employment status). It is likely that 
siblings, parents, spouses, and other relatives of caseheads are included in the NALS "AFDC" population. 
How this extended group's literacy profile compares with that of actual AFDC caseheads is uncertain. 



(1993), focusing on the literacy needs of younger welfare clients, reports that recipients 
between the ages of 17 and 21 typically have reading skills below "the 6th grade level". 
The recently completed National Evaluation of Adult Education, which examined a 
random national sample of English as a Second Language, Adult Basic Education and 
Adult Secondary Education programs and students (Development Associates, 1992, 
1993), estimated that 19% of students were currently receiving AFDC and/or SSI, and 
that 40% of the welfare recipients possessed skills below "high school entry level". Marsh 
et al (1990) administered the Woodcock-Johnson Psycho-Educational Battery to a 
random sample of 106 individuals receiving public assistance in Arkansas. Mean grade 
level scores reported for the basic skills achievement tests are 5.9 for Reading, 5.8 for 
Written Language and 5.6 for Math. 

Because low educational attainment is generally associated with low levels of literacy 
proficiency (Kirsch et al, 1993), it is difficult to say, looking at the studies cited above, 
whether low levels of educational attainment are the major barrier or whether literacy per 
se is also a barrier to self-sufficiency among welfare clientele. This is a matter of 
considerable practical importance, of course, because many adult literacy programs, in 
effect, are attempting to raise the adults' literacy proficiencies without necessarily changing 
their educational attainment. 

Figure 1 provides some additional perspective on the joint effects of education and literacy 
on welfare dependence. The figure is based on our own secondary analysis of the NALS 
household survey data 2 . The figure displays the estimated percentage of households 
receiving AFDC or other public cash assistance (excluding SSI) as a joint function of the 
respondents' educational attainment and assessed literacy proficiency 3 . Three levels of 
educational attainment are shown, according to the highest credential attained: no 
credential, a secondary credential (high school diploma or GED certificate), or a post- 
secondary credential (2 or 4 year college or advanced degree). The five literacy 
proficiency levels shown are the ones reported by Kirsch et al (1993) and adopted by the 
National Education Goals Panel for reporting adult literacy proficiency 4 . The bars in the 
figure do not provide information about the frequency of particular combinations of 
educational attainment and literacy proficiency in the adult population; they indicate the 
percentage of those having particular combinations of education and literacy who receive 
cash assistance 5 . Notice that there is no data shown for one particular combination: the 



2 The incarcerated component of the NALS sample is not included in this analysis for obvious reasons. To 
simplify interpretation, the data are further limited to individuals satisfying three conditions: (1) not 
currently enrolled in school, (2) born in the United States; and (3) spoke English before starting school. 
3 Footnote 1 discusses some limitations in interpreting these data. 

4 For simplicity, the three NALS proficiency scales (prose, document, and quantitative) are averaged here; 
results do not vary substantially with the particular scale utilized. Furthermore, the three scales are so 
highly intercorrelated that they can be well approximated by a single dimension of literacy proficiency 
(Reder, 1994a). 

5 Most of these AFDC clients are in fact in the lower levels of educational attainment and literacy 
proficiency. Of the AFDC population as qualified by Footnotes 1 and 2, 70% are in the lowest two 
literacy levels, 39% do not have a high school diploma or GED, and 35% are in the two lowest literacy 
levels and do not have cither a high school diploma or GED. 
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lowest educational attainment with the highest level of literacy. The lack of data in the 
figure does not mean that there is no cash assistance associated with this combination; it 
means that there are no adults in our society with that little education who have that high a 
literacy proficiency 6 On the other hand, the near-zero bar in the adjacent cell of the figure 
~ i.e., those with a secondary credential and the highest literacy level ~ means something 
quite different: Virtually no such individuals receive AFDC or other public assistance. 

Inspecting the figure, it is clear that both education and literacy appear to influence 
welfare dependence. At each level of educational attainment in the figure, increasing 
literacy is associated with decreasing rates of welfare utilization. At the same time, at 
each given level of literacy proficiency, increased educational attainment is associated with 
decreasing welfare dependence. Multivariate predictive models for welfare utilization that 
we have applied to these data, which take into account the effects of additional variables 
such as age, gender and employment history, confirm that both education and literacy are 
significantly associated with welfare utilization 7 . 

Do literacy programs increase the basic skills of welfare clients? Although there are 
innumerable reports that students enrolled in various types of adult literacy programs 
experience significant learning gains, there are severe methodological problems associated 
with the ways in which such gains are generally measured (Venezky, Bristow & Sabatini. 
1993). In addition, most learning gain data is collected while students are still in an 
instructional program, typically a pre-test near program entry and a post-test at 
program completion or exit or after a fixed period of enrollment. Improved 
performance between the pre-and post-tests often measures phenomena other than basic 
skill development, such as increased familiarity or recent experience with test-taking, 
"teaching to the test", and so forth (Metis Associates, 1991; Sticht & Armstrong, 1993; 
Venezky et al, 1993). Although many studies and program evaluations report learning 
gains, mosi; are subject to these problems and thus generally provide weak evidence of 
long-term learning gains among program participants. The recently completed National 
Evaluation of Adult Education, which attempted to estimate the effects of program 
participation on learning gains for a random national sample of programs and students, 
encountered many of these thorny technical problems and may not be able to provide 
convincing evidence that program participation substantially increases adults' basic 
skills (Fitzgerald & Young, 1994). 
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6 There are, in fact, a few such individuals in the sample, but not enough to reliably estimate their AFDC 
rate in the population. 

7 These analyses are not presented here because of the aforementioned uncertainty about the caseholder 
status of the survey respondents who indicated (someone in) their household received public assistance. 
See Footnote 1. 
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Figure 1. Percentage of households receiving AFDC or other cash welfare by adults' highest 
educational credential and assessed literacy proficiency. Source : Authors' analysis of National Adult 
Literacy Survey data Data are weighted and exclude adults currently enrolled in school, or born 
outside of the United States or who did not speak English before entering school. See Footnote 1 for 
cautionary note. 
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Another approach to studying the literacy development of program participants over 
time was taken by Martinson and Friedlander (1994) in their evaluation of the 
educational outcomes of California's Greater Avenues to Independence (GAIN) program 
(a JOBS program which adds a basic education component to the general welfare-to- 
work programs). The evaluation of GAIN'S educational outcomes was embedded in a 
larger evaluation study of the cost-effectiveness of GAIN in moving AFDC clients 
needing basic education from welfare to work. The larger evaluation study used an 
experimental methodology in which clients deemed in need of basic education were 
randomly assigned to one of two service streams. One group (called the 
"experimentals") was mandated to participate in a basic education program (GAIN),' 
whereas the other (termed the "controls") entered the general county welfare program. 

Clients' literacy proficiency was assessed during program intake with a standardized 
basic skills test. Clients without a high school education or sufficiently high pre-test 
scores were identified as needing basic education. Half of these individuals were 
randomly assigned to GAIN, the remaining half to the general county welfare-to-work 
program. Two to three years after random assignment, samples of experimentals and 
controls were paid to participate in a post-test of their basic skills. As expected with 
such random assignment, the pre-test literacy scores and other background 
characteristics of the two groups did not differ significantly. Differences observed 
between the two groups' post-test scores are thus presumably attributable to being 
randomly assigned to the experimental GAIN program. Welfare clientele from five 
California counties with GAIN programs participated in the evaluation of educational 
outcomes. Overall, there was no significant difference between the controls' and 
experimentals' post-test scores, indicating little systematic effect of participation in 
literacy programs on basic skills development. Although clients assigned to one of the 
county's GAIN programs who entered with relatively high literacy skills did have 
significantly higher post-test scores than comparable controls, there is little indication 
of a broad or consistent effect of instruction across groups of clients or county 
programs. 8 

There is thus little evidence yet available that convincingly shows that program 
participation substantially improves adult students' literacy proficiencies. 
Unfortunately, this is the case regardless of whether the learners are mandatory 
participants in welfare-to-work programs like GAIN or voluntary participants more 
typical of the nation's adult literacy programs. At the same time, there is much 
anecdotal evidence and widespread perceptions among adult literacy students and 
teachers that literacy development is as easy to find in adult education as it is difficult 
to detect with standardized tests. This non-alignment of perceived and objectively 
measured growth in adult literacy continues to perplex and frustrate, practitioners and 
policy makers alike. As will be seen below, many believe this discrepancy exists 



8 Later sections of this report discuss methodological problems that arise in interpreting the GAIN 
evaluation of educational outcomes, and how these problems limit the conclusions that should be drawn 
from these findings. 
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because standardized basic skills assessment instruments and procedures themselves are 
not aligned with what is taught and learned in adult literacy programs. 

What are the economic consequences of raising the literacy proficiencies of welfare 
clients? Research reviewed above clearly shows that welfare clients have markedly less 
education and literacy proficiency (even after adjusting for lower education). Although 
that pattern is certainly consistent with the notion that increasing clients' basic skills might 
decrease their overall dependence on cash assistance, such a conclusion does not 
necessarily follow. These correlational relationships do not imply that interventions which 
raise literacy skills, for example, would thereby reduce future welfare dependency. To 
establish that conclusion, two things need to be shown. First, it must be shown that some 
interventions do raise literacy proficiencies (a point which the preceding section stated is 
not yet soundly established). Second, it needs to be shown that if literacy scores do 
indeed increase, economic outcomes subsequently improve. 

There are very few, if indeed any at all, studies which unequivocally show that changes in 
assessed literacy abilities lead over time to changes in economic outcomes such as welfare 
dependence. The GAIN evaluation in principle was to have indicated how program- 
induced changes in clients' basic skills are associated with their changing patterns of 
employment, earnings and welfare dependency. Since no overall program-induced effects 
were found on literacy skills, it is, of course, not feasible to test for subsequent effects on 
economic outcomes. But other more indirect kinds of evidence are available which 
suggest that increases in welfare clients' literacy skills can increase their future earnings 
and thus hopefully decrease their future welfare dependency. 

Hughes (1991) reported results of a three-year tracking study of participants in Oregon's 
New JOBS program (the experimental forerunner of many JOBS programs in Oregon, 
including Steps to Success, the program examined later in this report) during the late 
1980s. The BASIS, a customized basic skills test, was administered to all clients entering 
the New JOBS program 9 . Their subsequent earnings, derived from SS-matched state 
unemployment insurance (UI) hour and wage records, were tracked for three years. 

Figure 2 displays the average annual earnings for those clients who had at least some UI 
earnings during each of the three tracking years. Clients are grouped in the figure 
according to their level of BASIS-assessed math skills at program entry. As Hughes 
(1991) notes, there is relatively little difference in average earnings among the skill-level 
groups during the first tracking year; with each successive year, earnings of the groups 
further diverge according to literacy level, with higher skilled clients earning progressively 
more than lower skilled clients. 

Hughes (1991) is careful to point out that these trends do not necessarily reflect program 
impact on the subsequent employment and earnings of welfare clients 10 . But the data 



9 More information about the BASIS tests is presented in subsequent sections. 

10 Hughes (1991) points out, for example, that clients may have gone through the New JOBS program at 
any time during the three year period, so that the tracking years do not necessarily follow the delivery of 
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certainly do suggest that literacy skills may well affect welfare clients' subsequent 
employment and economic status. A similar pattern, however, is seen, when clients are 
stratified according to their educational attainment, with the more educated clients 
showing increasing divergence in earnings across tracking years. Hughes did not attempt 
to disentangle the effects of education and literacy skills in these data. 
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Figure 2. Average earnings for three successive years of Oregon's New JOBS clientele, plotted by 
BASIS Math scores assessed at program entry. Data are for those clients who had some UI earnings 
in each of the three years. Source : Hughes (1991). 



Summary: Policy context for the research project. The evidence about the efficacy of 
mandating welfare clients to participate in basic skills education is thus mixed. There is 
abundant cross-sectional evidence which suggests that literacy (even with education held 
constant) is positively related to economic outcomes like employment, earnings, poverty 
and welfare status (Berlin & Sum, 1988; Kirsch et al, 1993). There is little evidence 
available, however, that indicates whether participation in adult literacy education leads to 
sustained increases in skills (i.e., learning gains that remain well after program 
termination). Methodological problems in most longitudinal studies (including the 
recently completed National Evaluation of Adult Education) limit the conclusions that can 
be drawn on this point. The GAIN study suggests that such sustained learning outcomes 
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program services. And there is no information about whether clients' assessed literacy skills increased 
during the tracking period. 
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may not be widespread when participation is mandatory, although the report's authors 
(Martinson & Friedlander, 1 994) caution that methodological factors may limit the 
generality of their findings as well. The present study, then, will try to fill in some of the 
gaps and uncertainties about how program participation leads to literacy development,„and 
how that literacy development in turn impacts clients' future economic success and 
personal growth. 



The Research Project; Literacy Development at Steps to Success 

Origins of the project This project grew out of two sources. It was practitioner- 
initiated in that project staff had been hearing informally from a broad network of adult 
education practitioners that the Career and Life Planning (CLP) component of the Steps to 
Success program seemed to markedly increase active participation and learning in the 
adult education courses which followed it. As described in the Dedication and Preface, 
the second source for the project was the prior work of project staff following a cohort of 
women as they moved through the CLP course at Steps (Wikelund, 1993). Wikelund's 
work itself grew out of the practitioners' perceptions of CLP's impact on adult education. 

The research setting: Steps to Success. Steps to Success is the largest JOBS program in 
Oregon. In operation since the fall of 1990, Steps serves welfare clients at two primary 
locations in Multnomah County within the Portland metropolitan area. Like other JOBS 
programs, Steps is a welfare-to-work program which provides a range of services to 
eligible clients to facilitate their transition to employment and economic self-sufficiency. 
Steps is administered by Mt. Hood Community College under contract with the Oregon 
Adult and Family Services Division. Its activities are coordinated with those of several 
state and local agencies, especially those providing employment, vocational rehabilitation, 
job training and other social services. Wikelund (1993) describes the program in detail. 11 

Welfare clients assigned to Steps must be screened by the program unless they have 
children under the age of one year or are otherwise deferred from active program 
participation. Depending on a range of factors - client characteristics and needs identified 
during intake screening and assessment, program service priorities and the availability of 
open service "slots" - Steps clients are assigned to one of two service tracks. Clients 
who are deemed to possess marketable skills are initially assigned to the Placement Track. 
Clients in the Placement Track participate in a customized mix of job training, search and 
placement activities. Clients whose intake screening and assessment indicate that they 
needed to enhance their basic literacy skills and/or vocational skills were initially assigned 
to the Career and Life Planning (CLP) Track, after acquiring needed skills in the CLP 
Track, clients are transferred into the Placement Track. 

The intake screening and assessment activities which determined whether clients were 
assigned to the CLP Track included a one hour background interview and the 



1 'Wikelund (1993) used the pseudonym Pathways to Progress to refer to the program. Program staff 
preferred to have the actual name of the program, Steps to Success, used in this report. 
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administration of a secure, standardized basic skills test, the Basic Adult Skills Inventory 
System (BASIS). The BASIS is a competency-based functional literacy test reported on 
two proficiency scales: Reading and Math. The BASIS is a customized instrument 
developed by the Comprehensive Adult Student Assessment System (CASAS) specifically 
for use by Oregon community colleges, employment and social service agencies. (Further 
details about this instrument are presented below.) Clients generally needed to score 220 
or above on the BASIS Reading scale for placement in the CLP Track; it was felt that 
those with lower scores would not be able to use the written materials involved. To give 
some idea of how Steps practitioners interpret a 220 score, a guideline called the 
BASIS/CASAS Scale Score Interpretation for Occupational Planning (Office of 
Community College Services, 1990) specifies that scores between 215 and 230 are "nearly 
employable", which is explained in the following way: 

Participants functioning between 215 and 230 are able to handle basic literacy tasks 
and basic computational skills in a functional setting related to employment. A 
minimum wage job is possible if carefully selected. They have difficulty following 
more complex sets of directions and are functioning below junior high level. (Office 
of Community College Services, 1990) 

The Career and Life Planning Track. Clients assigned to the CLP Track generally 
participated first in a Career and Life Planning (CLP) course whose length varied but 
averaged about 4 weeks (typically 4 days per week). The CLP class involved a mixture of 
personal counseling and support group activities, detailed basic and vocational skills 
assessment, and life skills instruction that focused on personal and career goal-setting, 
building self-esteem, problem solving, assertiveness training, communication skills and 
other content designed to facilitate *he transition to employment and future self- 
sufficiency. Wikelund (1993) describes the CLP course in detail. 

After graduating from the CLP class, clients deemed in need improved basic skills spent a 
period of time averaging 4 months in participating in literacy training (ABE/GED) 
available in an on-site center. Depending on their incoming BASIS scores, career goals 
and training needs, clients either prepared for the GED or attempted to improve and/or 
"brush up on" targeted basic skills. The average BASIS reading score of clients assigned 
to the CLP Track has been 242 (Wikelund, 1993), which is in the range at which adults 
are "functioning at a secondary level and have a high probability of passing the GED with 
some preparation." (Office of Community College Services, 1990). These ABE/GED 
training activities were offered in a learning lab environment, with both one-on-one and 
small group instruction available. Clients were required to participate in these activities a 
minimum of 15 hours per week. 

After completing these basic skills training activities, these CLP clients began participating 
in a variety of job preparation, search and placement activities. 
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METHODOLOGY 



Two principal research methodologies were used in this study. One was primarily 
quantitative, attempting to collect follow-up post-test and questionnaire data from a 
relatively large and statistically representative group of former CLP Track participants. 
The second methodology was primarily qualitative, attempting to collect richer, in-depth 
information about program impact and clients' post-program life outcomes from a 
relatively small and purposively selected group of former CLP Track participants. 

These approaches were designed to complement one another, so that, for example, the 
quantitative analysis of skill development evident in gains between pre- and post-program 
basic skills tests could be put into a richer and more readily interpretable framework with 
the perspectives of a cohort of former CLP Track clients with whom a member of the 
research team had previously developed close rapport and had studied before (Wikelund, 
1993). In order to link the two approaches closely, the latter group of clients was also 
invited to participate in taking the post-test and completing the activities questionnaire 
(although their test and questionnaire data were analyzed separately and are not reported 
as part of the overall quantitative data). In this way, the research team could examine the 
experience of the special cohort in taking the post-test and completing the questionnaire, 
giving in-depth insight about the test-taking process itself as well as about the perceived 
impact of Steps to Success on their lives. 

Quantitative Component 

Project staff worked closely with staff from Steps and the Office of Community College 
Services to design and implement the quantitative component. At planning meetings, 
issues and procedures were examined for identifying and contacting targeted program 
clients; scheduling a range of convenient times and locations for group administration of 
tests and questionnaires; administering the basic skills test and questionnaires; 
compensating individuals for taking the test and completing the questionnaire; quality 
control processes for collecting, scanning and data processing information on the 
standardized test answer sheets and questionnaire forms. 

The Mailing List: Target Participants. From agency databases, a list was constructed 
of Steps clients who had participated in an assigned ABE/GED instructional component. 
As described in the previous section, this group does not include all Steps clients, since 
many are not assigned to the CLP Track. In all, records of 920 appropriate Steps clients 
were entered into the initial project database. These records included: names and 
addresses for mailing invitations to participate in the study; demographic information from 
client records (age, gender, educational status, race/ethnicity); BASIS test scores from 
their intake assessment; and the status and duration of assigned Steps to Success program 
activities (including but not limited to the ABE/GED component mentioned above). 

The original design of the project envisioned comparisons of the literacy skill gains and life 
outcomes of those clients who took a Career and Life Planning (CLP) component prior to 
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taking an ABE/GED component with the outcomes of those who took the ABE/GED 
component but not the front-loaded CLP. In building and using the aforementioned 
database, however, it became clear that contrary to early indications, there were in fact 
very few clients who had taken ABE/GED without having first taken the CLP component. 
Unfortunately, it was therefore not feasible to make the planned comparison between 
clients who had and had not participated in CLP before taking the Steps basic skills 
training. 

First mailing. Letters to prospective study participants were mailed in mid-March, 1993 
to all targeted Steps "graduates" who had left Steps between 1 and 3 years prior to the 
mailing (with a median of 24 months). The form letter explained the purpose of the study 
and offered participants a choice of numerous 1-1/2 hour group sessions for coming in to 
take a basic skills test and complete a brief questionnaire. A stipend of $25 was offered to 
offset incidental expenses such as transportation and childcare and to motivate 
participation. Eighteen sessions were offered during April, 1993, at a range of times of 
day and days of the week (weekdays, weekday evenings and Saturdays) and at one of 
three locations spread across the Steps service area. Individuals who were interested in 
participating returned a postpaid reply card, indicating which session(s) they would like to 
attend. Confirmation cards were mailed to these individuals, which served as a ticket to a 
given session (quality control and stipending procedures precluded the acceptance of 
walk^ins). Copies of the letter, enclosure and reply card are contained in Appendix B. 

Second mailing. A second letter was mailed in late May, 1993, after sessions initiated by 
the first mailing had been completed. The second letter was accompanied by the same 
postpaid, self-addressed reply card enclosed with the first letter. The second mailing went 
to individuals on the original list who had not yet participated in the testing and whose 
first letter had not been returned by the post office as undeliverable. The second letter 
offered a choice of 6 additional sessions at three locations in June, 1993. A copy of the 
second letter is contained in Appendix B. 

Basic Skills Pre-Test. The pre-test scores used were the previously described BASIS 
Reading and Math scores from the clients' intake assessments. As noted above, the 
BASIS is a secure, competency based functional literacy assessment instrument developed 
by CASAS expressly for use by educational, employment and social service agencies in 
Oregon. The customized BASIS tests are derived from CASAS Rasch-scaled item banks, 
and are comparable with CASAS tests being used in other JOBS programs and adult 
education programs across the country. CASAS-scaled basic skills tests are presently 
being used in statewide JOBS programs in five states (CA, CT, KS, NC, OR) and locally 
in some JOBS programs in at least six other states (CO, ID, IN, MD, MI, & WA). Within 
Oregon, the BASIS is being used extensively by a range of adult service providers, 
including 1 1 of the state's 16 community colleges as well as in programs operated by the 
Corrections, JTPA, Adult and Family Services, Displaced Workers Program, Employment 
Division and other state agencies. 
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Appendix C provides more detailed information about the BASIS test and the 
interpretation of the CAS AS scale scores that it reports. Here is an overview of how 
BASIS/CASAS scores are interpreted and used within Steps: 

• Level A: < 200: difficulty with basic literacy & computation needed to function in 
workplace & community 

• Level B: 200-215: low literacy skills, making it difficult for all but entry-level 
employment 

• Level C: 216-230: has basic skills needed for basic functional tasks related to 
employment 

• Level D: 231-235: (Pre-GED) - could benefit from preliminary GED preparation 

• GED: 236+ : Functions at secondary level - high probability of passing the GED with 
some preparation 

According to the multi-agency statewide BASIS database, the average BASIS score of 
Oregon adults reporting an 8th grade education is 235 for Reading and 213 for Math 
(Employment Division, 1993). 

Basic Skills Post-Test. The CAS AS Levels Reading and Math (Level C) were selected as 
the most appropriate instruments to use for post-testing purposes in this study. The 
BASIS and CASAS tests are built from the same bank of test items, commonly scaled 
according to one parameter Item Response Theory (one scale for Reading, another for 
Math). Standard CASAS test booklets (Form 37R for Reading; Form 35M for Math) and 
scannable answer sheets were utilized. 

Test Administrators. Three staff from Steps to Success, previously trained and certified 
to give BASIS tests,, were hired to administer the scheduled CASAS testing/questionnaire 
sessions. One Test Administrator was assigned to each given session. Test 
Administrators were trained to utilize the test administration procedures described below. 
Test administrators were also debriefed by project staff after the completion of the final 
testing session. The debriefing focused on their perceptions of the reactions of Steps 
graduates to taking the basic skills test, significant comments they heard from participants, 
and related matters. 

Test Administration Procedures. The written procedures and instructions for 
administering the testing/questionnaire sessions are listed in Appendix B. Essentially, 
project staff prepared advance rosters of participants for each scheduled session, and used 
couriers to send the requisite materials to each session (a quality control sheet and 
participant roster, test booklets and answer sheets, pens, pencils and questionnaires, 
stipend receipts and stipend checks) to the administrator, who would return the completed 
control sheets, signed receipts, answer sheets and questionnaires to project staff. 

Sessions averaged about 1-3/4 hours in length, and includ;d the administration of the 
CASAS Level C Reading (Form 37R) and Math (Form 35M) tests, completion of the 
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questionnaire and stipend receipt form. Forty minutes were allowed for each test 12 . A 5- 
10 minute break was allowed between the two test taking periods. The remainder of the 
session was devoted to general instructions, completing the written questionnaire and 
stipend receipt, and to distributing the stipend checks. 

Questionnaire. A brief written questionnaire was developed for use in the sessions that 
gathered information about participants' activities and perceptions, including: (1) current 
educational and employment status; (2) additional education and training taken since 
leaving Steps; (3) retrospective assessment of how participating in Steps had helped them 
in various areas of their lives; (4) whether (and if so, how) Steps might have better 
prepared them; and (4) perceptions of improvements if any in their basic skills. The 
specific areas in which participants were queried about whether Steps had "made a 
difference" in their lives were taken from the program's- own goals or were areas which 
Wikelund's (1993) cohort had indicated were relevant to their social and economic well- 
being: getting a job; becoming self-sufficient; gaining control over their own lives; and 
being able to assist their children. Participants were asked to rate (on a three-point scale) 
the extent to which their basic skills had improved in reading, math, and writing. A copy 
of the questionnaire is in Appendix D. 

Response to the mailing and turnout for the testing and questionnaire sessions. 
Letters were mailed to the targeted 920 former Steps to Success clients. The post office 
returned 109 of these 920 letters as undeliverable. Of the remaining 811 individuals 
whose letters had not been returned by the post office, 229 contacted the project office to 
sign up for a session, an effective response rate of 27%. Since the actual number of "dead 
letters" in the mailing (i.e., those that were not forwarded to or received by the intended 
recipient) was probably much higher than the 109 officially returned by the post office, the 
effective response rate was also very likely much higher than 27%. Considering that some 
individuals had left Steps more than 3 years before the mailing, the overall response rate to 
the invitational mailing seems quite satisfactory. 

Even though individuals preregistered for specific sessions, and received confirmation 
notices of their registration as reminders, there was a high rate (averaging about 40%) of 
no shows at the sessions. Many of these individuals recontacted the project office, often 
with accounts of how child care, employment or transportation problems prevented them 
from attending the designated session 13 . Individuals were permitted to reschedule and 
participate in subsequent sessions. Many did so. 
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12 The CASAS Level tests were designed for hour-long test periods rather than the 40 minutes allowed in 
this study. It was not feasible to increase the length of the sessions to accommodate these longer periods. 
However, inspection of the completed answer sheets and debriefing sessions with the test administrators 
and the focus group indicated that very few if any participants found the time allotted to be problematical. 
13 Interestingly, the problems adults experienced in trying to participate in these activities are very 
reminiscent of those experienced more generally by adults trying to participate in adult basic education 
(Beder & Valentine, 1990; Wikelund, Reder & Hart-Landsberg, 1992). 
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A total of 163 of the 229 individuals who contacted the project office appeared for a 
scheduled testing and questionnaire session. Of those 163 participants, 112 (69%) chose 
sessions held on weekends or weekday evenings, and 5 1 (31%) participated in weekday 
daytime sessions. 

"Hard" outcomes. Two "hard" outcome measures were supplied by agency staff for 
each of the 163 individuals who completed the testing and questionnaire: (1) A binary 
indicator of whether the individual received cash assistance (AFDC) in Oregon during July 
1994; and (2) the individual's earnings reported by employers for unemployment 
insurance (UI) purposes 14 for the quarter in which the testing took place (i.e., the 2nd 
quarter of 1993). 

Data Processing. All CASAS Reading and Math answer sheets - on which project staff 
had precoded session identifiers and booklet form numbers as quality control steps — were 
delivered to a local community college for optical scanning, scoring and scaling with the 
use of CASAS scoring software. Results were returned on diskette. Questionnaire data 
were entered directly into machine readable files for processing. Unique client identifiers 
were used to merge results from the optically scanned and scored test data, questionnaire 
data and Steps to Success client information files. Further data processing and statistical 
analyses were conducted with SPSS for Windows, Version 6.1. 



Qualitative Component 

The longitudinal cohort. The qualitative component of the research consisted of several 
information gathering activities. Most of these research activities were grounded in the 
ongoing informal contacts between a member of the research team and a cohort of 
previous Steps clients who were involved in her ongoing research (Wikelund, 1993). The 
original contact with the cohort of 27 women occurred in an intensive four-week CLP 
class at Steps, in which the researcher was an active participant-observer along with these 
27 women. Fifteen of the 27 women were subsequently involved in in-depth ongoing 
research as they continued through and eventually graduated from Steps. The 
characteristics and previous experiences of these women have been described in detail 
elsewhere (Wikelund, 1993). 

Cohort post-tests and questionnaires. Although informal contacts with the cohort 
provided rich information about their ongoing lives - including changes in their welfare, 
employment and family status - information was needed which would help link the 
experiences and life outcomes of the cohort to the information supplied by the study 
participants through the post-tests and questionnaires. To establish a base of information 
common to the cohort and the participants in the quantitative study, cohort members were 
also invited to participate in the post-test and questionnaire activities. 



14 Based on wages and hours reported by employers for state unemployment insurance purposes. 
According to Hughes (1991), approximately 90% of Oregon jobs are covered by this system. Military 
employment, self-employment and out-of-state employment are not include 
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Cohort members were contacted by mail and offered the chance to participate in the 
follow-up testing and questionnaire activities much in the same way as other former clients 
of Steps. They were invited to come to a specific testing and questionnaire session. Only 
cohort members were invited to this particular session (i.e., it was not available to other 
study participants). The same testing and questionnaire administration procedures were 
used (in fact, the test administrator assigned to the session was not told that this was a 
"special" session). 

The cohort focus group meeting. Following the testing and questionnaire session 
cohort members participated in an informal focus group session to discuss their experience 
and perceptions of taking basic skills test and completing the questionnaire. Their close 
rapport they had with a member of the research team was thought to be an excellent 
context for this discussion to expand into a richer discourse about the successes and 
challenges they had encountered in their ongoing lives and attempts to transition 
successfully into self-sufficiency. Immediately following their testing and questionnaire 
session (which was held on a Saturday morning), an informal luncheon "reunion party" 
(they had not been together en masse since leaving Steps more about a year before) was 
held next to the testing room, Members of the research team facilitated the ensuing semi- 
structured but energetic and free-wheeling discussion. The group provided very rich 
descriptions and insights about the process of taking the test and completing the 
questionnaire. Many valuable comments were contributed about how the information 
collected by such standardized tests and questionnaires does and does not reflect what 
they had learned in Steps and the impact they felt Steps had had on their lives. 

Cohort participation in the testing, questionnaire and focus group activities. Twelve 
women from the cohort participated in the testing and questionnaire session and the 
ensuing "focus group" luncheon. As planned, their test results and questionnaire data are 
not included in the quantitative results presented in this report 15 . By participating in the 
test-taking and questionnaire-completing activities, these cohort members were "primed" 
to participate in the immediately following focus group discussion about the testing and 
survey questionnaire processes and the related issues about their continuing experiences 
with basic skills and education as they impacted their lives in general. Although their post- 
test scores and questionnaire responses are not included in the larger sample being 
analyzed here, the plan was to use these supplementary data in assembling a qualitative 
profile of the cohort, updating Wikelund's (1993) initial profile. 
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15 For several reasons, it was decided in advance not to include their test scores and questionnaire 
responses in the quantitative study. First, despite efforts to treat cohort members in much the same way as 
other study participants in terms of recruitment and administrative procedures (e.g., soliciting their 
participation by mail, offering the same stipend ), the extensive prior contact of the cohort with project 
staff created a distinct context for their engagement in these follow-up activities. Furthermore, the cohort 
members had gone through the CLP Track of Steps to Success more recently than any of the target study 
participants. 
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Debriefing meeting with the testing & questionnaire session administrators. After all 
testing/questionnaire sessions had been completed, in-depth discussions were held with 
each of the three staff administrators. These highly informal discussions were conducted 
both individually and as a group. The content of the discussions ranged widely, but 
focused mainly on: (1) the administrators' perceptions, impressions or comments about the 
testing and questionnaire activities; (2) which "kinds of former clients tended to 
participate in these activities; and (3) anecdotal accounts of participant comments, 
reactions or attitudes towards the tests, the questionnaire, Steps to Success, and so forth. 

Ethnographic research with the cohort. Ongoing contacts, informal interviews and 
other qualitative research activities were also taking place with the cohort members on an 
individual basis, so as to maintain an up-to-date profile of how their lives were progressing 
and how their basic skills were developing in relation to the challenges and opportunities 
they confronted. The basic skills post-test scores and questionnaire data Were to be 
included in this updated profile of the cohort. Unfortunately, as described in the Preface 
and Dedication to this report, these rich and invaluable research activities could not be 
completed or brought to closure because of the untimely illness and death of Karen 
Wikelund. 

Workshop presentations to staff from Steps to Success and other JOBS and adult 
education programs. A workshop was developed for presentation to staff of several 
types of adult literacy programs. The hour-long workshop was presented several times 
during the course of the project: to an all staff meeting at Steps to Success; to a meeting of 
all JOBS program managers in the region; and to a meeting of all Adult Basic Education 
Directors from community colleges across the state. Each audience consisted of about 25 
individuals, and provided very lively feedback and discussion regarding the findings of the 
study and their implications for literacy education in other programs. Their comments and 
suggestions helped guide the interpretation of the study's results in much the same way as 
other study participants in terms of recruitment and administrative procedures. 
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RESULTS 



Characteristics of Study Participants 

Statewide, countywide and target program client characteristics. As noted in the 
foregoing Methodology section, the target population for the study consisted of all Steps 
to Success clients who completed a basic skills education component within Steps. To see 
how this target population compares with other welfare client populations in the state, its 
characteristics can be compared with those of two other populations: (1) all welfare clients 
in the state of Oregon; and (2) all welfare clients in Multnomah County (where Steps 
operates). Selected characteristics of these three groups are displayed in Table 1. Data in 
the table are drawn from the Oregon statewide BASIS database as of April 1993 
(Employment Division, 1993). Information about Steps clients activities (upon which the 
targeting was based) are drawn from the Steps local JOBS Automated System (JAS) 
database. 

TABLE 1 

Selected Characteristics of Welfare Clients in Statewide, Countywide and Target 

Program Populations 



Characteristic 


State 
Welfare 
Clients 


County 
Welfare 
Clients 


Steps 
ABE/GED 
Clients 


N 


27555 


6678 


920 


% Women 


76.2 


81.2 


93.0 


% Minority 


19.3 


35.1 


21.1 


A * 

Age 


29.5 


30.0 


29.5 


Years of schooling 


10.9 


10.9 


10.7 


BASIS Reading Score 


242.1 


240.7 


239.9 


BASIS Math Score 


223.4 


221.1 


217.7 



*Age is reported in subranges (e.g., 18-20, 21-24) in the statewide BASIS database. The mean ages 
listed are estimated from the distributions of the age categories. 



Note that there is an inclusive relationship among these populations. The statewide 
database of welfare clients, of course, includes those from Multnomah County, which in 
turn includes all of the Steps to Success clients. The table indicates that these three 
populations differ to a certain extent in terms of their demographic composition and 
overall BASIS-assessed literacy abilities 16 . Although more than three quarters of the 
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16 Statistical tests of differences among these groups' characteristics are not performed here, since there is 
no particular interest or meaning associated with the outcome of these comparisons. There is little reason 
to suppose, for example, that welfare clients in the most populous and urban county in the state would be 
representative of the state's welfare population as a whole. Similarly, the selective screening involved in 
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welfare clientele in the state database are women, the proportion of female clients is 
somewhat higher for Multnomah County and is substantially higher for the basic education 
clients in Steps to Success (93% of whom are women). The preponderance of women in 
the Steps CLP Track is a direct reflection of the program's design and service priorities. 
Multnomah County, Oregon's most urban and demographically diverse county, has a 
substantially larger percentage of minority welfare clients than does the state as a whole. 
The proportion of minority clients participating in the basic educational services offered by 
Steps to Success is considerably lower than that of Multnomah County's overall welfare 
clientele. Presumably one major reason for this is that Steps primary location for service 
delivery is relatively distant from the neighborhoods where many minority clientele live 17 . 
Neither age nor years of schooling appears to differ appreciably.among the three groups. 
Welfare clients' average BASIS reading and math scores are lower in Multnomah County 
than in the state as a whole, and still lower among the Steps clients assigned to the basic 
education activities. Given the screening and selectivity (which includes consideration of 
BASIS test scores) that goes into assigning Steps clients to the CLP Track, it is not 
surprising that their BASIS scores would differ appreciably from those of the overall 
county welfare clientele. 

Target population and realized sample. As described above, the realized sample of 163 
study participants volunteered to take part in the follow-up testing and questionnaire 
sessions. They responded to a mailing that offered a financial incentive to participate, 
which was sent to all previous Steps clients who had participated in an assigned 
ABE/GED activity. Table 2 compares the characteristics of the 163 study participants 
(the realized sample) with those of the remaining 757 members of the target population 
who did not participate in the study. (Note that the two groups together are the same as 
the group of 920 target Steps clients displayed in the Table 1 above.) 

Independent-samples t-tests were carried out to compare the characteristics of these study 
participants and non-participants. The resulting t-values and p-values from these tests are 
shown in the table. The p-value shown is the probability of observing the sample t-value 
under the null hypothesis that there is no difference between the two groups. At the .05 
significance level or beyond, all differences between the groups are statistically significant 
with the exceptions of (1) how long ago clients went through the CLP component at 
Steps, and (2) the marginal exception of years of schooling 18 . There is a higher proportion 
of women and a lower proportion of minorities in the realized sample than in the target 



assigning Steps clients to particular service tracks such as CLP/basic education itself should generate 
specialized distributions of client characteristics (Wikelund, 1993). 

17 There is a smaller Steps facility that is located closer to minority clientele. But other factors may also be 
involved in the low minority participation rate in the basic educational services provided by Steps to 
Success. 

18 Before conducting a given t-test, a preliminary Levene test was carried out to determine if the variances 
of the two groups are equal. The probability that the two variances are not equal given the sample data is 
shown in the rightmost column of Table 2. If the variances of the two groups were significantly different 
at the .05 level, then the t-test results shown in the table were predicated on the two distributions having 
different variances (this happened only for two variables: Years of schooling and the BASIS Reading 
Score). Otherwise, the t-test was carried out assuming the two groups had homogeneous variances. 
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population for the study. Almost all (97.6%) study participants are women, and 12.9% 
are members of minority groups. Because of the small size of the realized sample 
(N=163), it will not be possible to consider the results of the study separately by gender or 
minority status. Study participants were also about 2 years older (3 1 vs. 29) than non- 
participants and had BASIS scores 4 to 5 points higher than the non-participants, even 

TABLE 2 

Selected Characteristics of Study Participants and Non-Participants 



Characteristic 


Non- 
Participants 


Study 
Participants 


t-value 


p-value 


Levene equal- 
variance test 


N 


757 


163 








% Women 


91.9 


97.6 


3.55 


.000 


.000 


% Minority 


21.1 


12.9 


2.73 


.007 


.000 


Age 


29.1 


31.3 


3.74 


.000 


.027 


Years of Schooling 


10.7 


11.0 


. 1.92 .- 


.055 


.067 


BASIS Reading Score 


239.1 


243.7 


4.65 


.000 


.527 


BASIS Math Score 


217.0 


221.1 


4.05 


.000 


.031 


Months since CLP 
entry* 


23.0 


23.8 


1.61 


.109 


.350 



*Months since assignment to Career & Life Planning track of Steps to Success (as of May 1993) 

though the years of schooling did not differ significantly between the two groups. There 
may well have been effects of gender, age, minority status and skill-level in the self- 
selection of study participants from the target population. It is possible, for example, that 
individuals who do not score well on tests are more test-averse and were less likely to 
volunteer to participate in the study when presented the opportunity to do so. It is also 
possible that individuals' employment status and/or welfare status at the time participants 
were being recruited affected their schedules and resources in ways that made them more 
or less likely to participate. Such questions, of course, must be addressed when we 
consider the results of the study and ask how they might generalize from those 
participating in the study to the larger population of welfare clients. So we will return to 
these questions after presenting the results for the study participants. 



Learning Gains 

Issues in comparing pre- and post-test scores. Although both the BASIS instruments 
(used as pre-tests in this study) and CASAS Level Tests (used as post-tests for this study) 
measure math and reading proficiencies on the same IRT-calibrated scale, some technical 
problems arose in trying to compare their scores to assess the learning gains made by 
Steps clients. The BASIS tests were designed for use as screening instruments, measuring 
proficiencies over a relatively narrow "accurate range" (i.e., the range of scores which can 
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be imputed with relatively low standard errors) 19 . The CASAS Level C tests used in the 
study have broader accurate ranges. The relatively narrow accurate range of the pre-tests 
proved to be more of a practical problem with the reading test than with the math test, 
because most of the study participants actually tested outside of the designated accurate 
range of the BASIS Reading test, whereas most scored within the accurate range of the 
BASIS Math test. One technique suggested by CASAS for comparing scores in such 
circumstances is to eliminate individuals from the analysis who scored outside of the 
accurate range on either the pre- or post-test. Only 53 of the 163 individuals tested 
scored inside the accurate ranges of the Reading tests, a number too small to utilize for 
purposes of many of the statistical tests and relationships to be examined. 

Before leaving the reading score data, however, it should be noted that they do show clear 
evidence of overall learning gains. The reading scores of these 53 individuals average 
231.3 on the pre-test and 235.5 on the post-test, for an average gain of 4.2 scale points. 
A paired samples t-test indicates this gain to be significantly non-zero: t = 4.54, p < .001. 
Of the 53 individuals, 37 showed individual learning gains, 3 no difference, and 13 
exhibited negative gains. The median reading gain was 4.0 CASAS scale points. 

The math test scores present fewer technical problems to work with than the reading 
scores. Most (127/163) individuals scored in the accurate range of both the pre-test and 
post-test. Although the standard error of estimating an individual's proficiency is larger 
when outside of the instrument's accurate range, it is still preferable here to work with the 
entire sample in order to have a larger N for multivariate analyses involving test scores and 
other individual characteristics. As a check on whether including these outlying scores 
significantly biases the resulting learning gain outcomes, initial analyses were conducted 
with both the entire group of 163 individuals as well as with the subgroup of 127 falling 
within the accurate range. Table 3 displays summary statistics for the scores and math 
learning gains of the two groups. 



Table 3 
Math Learning Gains 



Sample 


N 


Pre- 


Post- 


Gain 


t 


P 


Gains 


Gains 


Gains 






Test 


Test 








>0 


= 0 


<0 


Entire group 


163 


221.1 


229.0 


7.9 


11.1 


.0001 


135 . 


5 


23 


"Accurate range" 


127 


217.9 


226.7 


8.8 


13.7 


.0001 


113 


5 


9 



Results for the entire sample and the subsample falling within the accurate range are 
shown in the bottom two rows of the table. Overall results appear quite similar for the 
two samples, although scores are lower for the entire group because more individuals 
scored below the accurate range than above it. The important point to note, however, is 



19 Thanks to Dr. Patricia Rickard of CASAS for providing additional technical information about using 
and interpreting the BASIS and CASAS tests used in this study. 
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that each group has a large learning gain, averaging about 8 scale points for the entire 
group or 9 scale points for the subgroup. In either sample, a paired t-test comparing pre- 
test and post-test scores is statistically significant at the .0001 level or better. As the 
rightmost three columns of the table show, the vast majority of individuals experienced 
positive learning gains using either sample. 




Pre-test Score 



Figure 3. Relationship between math pre-test (BASIS) and post-test (CASAS Level C) scores. 
Each point represents an individual; heavier points represent the coincidence of multiple 
individuals. Points above and below the diagonal line represent positive and negative learning 
gains, respectively. Pearson correlation between pre-test and post-test is .65 (N=163). 

Figure 3 displays the relationship between individuals' pre- and post-test math scores. 
Ench point represents an individual. Heavier points represent the coincidence of two or 
more individuals. The diagonal line represents equal performance on the two tests. Points 
above the diagonal line represent positive learning gains, whereas those below the line 
represent negative gains. As noted above in Table 3, the vast majority of these Steps 
clients experienced positive learning gains. This figure indicates that these positive 
learning gains occurred throughout the proficiency scale; clients coming into Steps at a 
broad range of proficiencies benefited from the program and increased their math skills. 
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Math Learning Gain 



Figure 4. Frequency histogram of math learning gains for the 163 study participants. Gains are 
expressed in CASAS scale points. 

Figure 4 shows a frequency histogram of these math learning gains. Most of the 
distribution is centered to the right of zero, since nearly all individuals exhibited positive 
gains. Although it was noted above that these already impressive gains are slightly larger 
in the subgroup scoring within the accurate range of the tests, data from the entire group 
of 163 participants will be used in subsequent analyses 20 . 

Features of learning gains. There are several noteworthy features of these learning 
gains. First, in terms of their size, the reading gains (averaging 4 CASAS scale points) are 
typical of the CASAS gains reported between pre-test and post-tests given at program 
entry and exit. The math learning gains (averaging 8 scale points) are somewhat larger 
than those typically reported in the usual pre- and post-test comparisons within adult basic 
education programs (California Adult Student Assessment System, 1993a; Development 
Associates, 1994; Fitzgerald & Young, 1994; Iowa Department of Education, 1994; 
Office of Community College Services, 1994; Sticht & Armstrong, 1993). That the Steps 
clients' math gains are larger than their reading gains may well reflect their lower entering 
skill levels in math and the broad perception among Steps intake counselors and adult 
education teachers a greater need for improving clients' math skills. Teachers reported 



20 Although not included in the analyses, it should be noted that among the 12 women from Wikelund's 
(1993) cohort who partiicpated in the testing, the average gain in math scores was 10.4 points; 1 1 of the 
12 achieved individual gains, and 1 had the same pre- and post-test scores. 
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placing more emphasis on improving math skills, often appealing to the close linkage of 
math skills to the workplace 21 . 

The average math pre-test and post-test scores (see Table 3) correspond in Oregon adult 
education norms to grade levels of 10.0 and 12.3, respectively (Employment Division, 
1993). Among statewide welfare clientele, the corresponding grade level equivalents are 
9.9 and 12.4, respectively 22 . Thus these math gains, which span about 2-1/2 grade level 
equivalents, bring the average math skills of the Steps clientele above the level of those 
with high school diplomas 23 . Gains of this magnitude are well beyond those typically 
found in careful studies of pre- and post-test scores in basic skills programs, regardless of 
the duration of instruction or the types of assessment instruments used (Sticht & 
Armstrong, 1993). 

A second noteworthy feature of these data is that they represent sustained learning gains. 
Most learning gain data is collected while students are in an instructional program, 
typically a pre-test at program entry and a post-test at program completion or exit or after 
a fixed period of enrollment. Collected in this way, an improvement in performance 
between the pre-and post-tests often measures phenomena other than learning, such as 
increased familiarity or recent experience with test-taking, "teaching to the test", and so 
forth (Metis Associates, 1991; Venezky et al, 1993). The learning gain data from this 
study, on the other hand, are based on post-tests conducted approximately one to three 
years after exiting the instructional program. Not only are the math gains larger than 
usual, they are sustained learning gains. 

Related to the feature of sustained learning gains is the important question of retention of 
learning (Wagner, 1994). Clearly the sustained gains evident here represent a 
considerable retention of skills and knowledge learned. But to what extent has some of 
was originally learned in the program faded with the passage of time since exiting the 
program? To examine these possible retention effects, we examined the relationship 
between the elapsed time since completing Steps (based on computerized client records) 
and the learning gain. There was no statistically significant correlation between the two (r 
= -.123, n = 15 1, p = . 13 1). The negative sign of the correlation coefficient indicates that, 
as expected, smaller gains are associated with increasing time between the end.of the basic 
skills instruction and the post-test. But the effect is insubstantial and is not statistically 
significant. 



21 Although Wikelund directly observed a greater emphasis on math in the Steps class she observed, no 
direct observation was made of the classes which these study participants went through. The Steps 
program staff reported a greater emphasis on math consistently throughout the life of the program. 
22 These grade level equivalents are based on analysis of Oregon statewide BASIS database of 68,399 
clients tested by seven major state agencies or programs providing or referring to adult basic education 
services: Adult and Family Services (including JOBS), Alternate Schools, Department of Corrections, 
Dislocated Worker Program, Employment Division, Office of Community College Services, and Job 
Training Partnership Act. 

23 The comparison with those holding high school diplomas, of course, is limited to clients of the state 
agencies identified in the previous footnote. Comparable data are not available for a representative 
sample of high school graduates within the general state adult population. 
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A final feature of the learning gain data concerns the relationship of learning gain to the 
amount of basic skills instruction provided by Steps. The Steps database did not record 
the amount of instruction individuals received, merely the duration in calendar days during 
which an activity such as adult basic education was officially "active" in the client's case 
management records. In practice, this elapsed time proved to be a poor indicator of the 
amount of instruction received. So study participants were asked on the questionnaire to 
estimate the number of weeks of basic skills instruction they received in Steps (along with 
the number of days per week and the number of hours per day) 24 . The number of weeks 
of instruction was correlated with the assessed math learning gain. A statistically reliable, 
but weak positive correlation is found between the two (r = .243, n = 1 1 1, p = .01) 



GED Attainment 

A frequent goal in adult education for those who do not have a traditional high school 
diploma is attaining an alternative credential, such as the GED certificate. GED 
attainment is also a common Objective in JOBS programs for those whose entering basic 
skill levels indicate they might be able to prepare for and pass the GED Tests within a 
relatively short period of time (Martinson & Friedlander, 1994). In the California GAIN 
Project, for example, GED attainment is one of two targeted outcomes for clients' basic 
education (the other being increased literacy skills). The GAIN educational evaluation 
found program-enhanced rates of GED attainment to be much more widespread than 
program-enhanced increases in basic skills (Martinson & Friedlander, 1994). That pattern 
is consistent with the fact that GED attainment frequently may represent the certification 
of skills that adults already possess rather than the enhancement of their skill levels per se 
(Kaplan & Venezky, 1993). A detailed study of GED recipients in Oregon found (for a 
random sample of the state's adult population) that GED attainment appears to involve a 
mixture of skill certification and skill enhancement effects (Reder, 1994b). 

Of the 79 study participants who entered Steps without a high school diploma or 
equivalent, 28% (22 individuals) subsequently received a GED certificate during the 
follow-up period. This GED attainment rate is much higher than the rates Martinson and 
Friedlander (1994) report in their evaluation study of the educational outcomes of 
California's GAIN program. The overall GED (or high school diploma) attainment rate 
for those assigned at random to the various GAIN programs was 9% during a comparable 
follow-up period 25 . The Steps program thus appears quite effective in assisting clients to 
attain GED credentials. 



24 There were numerous missing data on the questionnaires regarding the duration of the basic skills 
instruction, especially from those individuals who exited the program 2-3 years earlier. The test 
administrators and members of the cohort focus group reported that many participants encountered 
difficulties recalling details about when various activities began, ended or how long they lasted. The 
validity of the date and duration responses in the questionnaires is therefore uncertain. 
25 This comparison is problematical on any number of grounds. First, the entering CASAS-scaled skill 
levels of participants in the basic education programs were not parallel between GAIN and Steps.. 
Although their math scores were relatively comparable, the reading scores of Steps participants were 
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Questionnaire Data 
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Each of the 163 participants in the follow-up testing completed the brief written 
questionnaire. The questionnaire collected information about participants' educational 
attainment (which served as an update to the educational status information provided at 
program entry), current employment status and any education and/or training received 
after leaving Steps. The questionnaire also probed participants' retrospective perceptions 
of Steps: Whether and how it impacted various aspects of their lives (e.g., finding work; 
helping their children; becoming more self-sufficient; attaining a sense of control over their 
lives); whether and how Steps could be improved; and the extent to which they felt their 
participation in Steps improved their basic skills (math, reading, writing). 

Perceived program impacts. Figure 5 displays the percentage of participants who 
reported that Steps helped them to improve various aspects of their lives. 
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Figure 5. Participants' perceptions of program impact. 



substantially higher, so that less additional skill growth should be needed for GED attainment. Indeed, 
Martinson and Friedlander (1994) reported thai the entering skill levels of GAIN clients had a significant 
impact on their rate of subsequent credential receipt. Second, as discussed in greater detail below, the 
"experimentals" who were randomly assigned to GAIN included a mix of actual participants and non- 
participants in GAIN'S "mandated" educational programs. It is thus quite likely that the 9% figure 
underestimates the GED attainment rate for actual program participants. 
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Participants' retrospective assessment of whether Steps to Success "made a difference" in 
their lives varies with the intended program impact. Although most participants (87%) felt 
that the program increased their self-sufficiency, slightly less than half (49%) perceived the 
program to have helped them find work. More than three-quarters (77%) reported that 
Steps made a difference in "controlling my life", and nearly two-thirds (64%) felt it had 
helped them to assist their own children. 

Employment and earnings. More than one in three (38%) of the study participants 
reported being employed on the questionnaire, of whom 57% reported working full-time 
and 43% part-time. The corresponding "hard data" for employment is highly consistent 
with this self-reported data. Overall, 48% of the participants had some earnings reported 
for unemployment insurance (UI) purposes during the 2nd quarter of 1993 (i.e., the 
quarter encompassing the testing/questionnaire sessions). This figure is a good indicator 
of the percentage of individuals who worked at some point during that 3 month period, 
which is expected to be higher than the percentage (38%) who reported themselves 
working at a given point of the period. Eighty-six percent (86%) of those who did report 
working at the time of the questionnaire had "hard" earnings during the quarter, which is 
close to the 90% estimate of jobs covered by the state unemployment insurance system 
(Hughes, 1991). 

The Ul-reported earnings for the second quarter of 1993 (during which the testing took 
place) ranged between $0 and $5382. The mean and median quarterly earnings of all 
study participants were $899 and $0, respectively. For those having some earnings during 
the quarter, the mean and median quarterly earnings were $1879 and $1649, respectively. 

Reflecting on these data, it is clear that Steps to Success graduates who chose to 
participate in the follow-up testing encompassed a broad range of employment and income 
situations at the time the test was given. Individuals with no earnings during that quarter 
(many of whom were likely to have been drawing public assistance) participated, as did 
individuals with fairly substantial incomes (who* likely were not receiving cash assistance at 
the time). 

Additional education and training. Two-thirds (67%) of the respondents reported 
having taking additional education or training after leaving Steps to Success. When asked 
to detail the education or training they had subsequently taken, various vocational and 
technical fields were mentioned, particularly the clerical, office support and medical 
technology fields. Very few mentioned any kind of additional basic skills education or 
GED preparation. 

Overall program effectiveness. When asked if Steps could have "better prepared" them, 
two-thirds (66%) indicated "no". Those who indicated that Steps could have better 
prepared them were asked to elaborate what additional help would have benefited them. 
Most of the open-ended responses mentioned needing "more time" in the program, not 
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being "pushed out into the world", "more time for schooling", additional assistance with 
career/life planning, and other activities prior to entering the labor force. 

Perceived basic skill development. The questionnaire asked participants to rate the 
extent to which they felt their basic skills improved as a result of participating in the 
ABE/GED classes at Steps. Ratings were given on a three-point scale ("No change", 
"Improved somewhat", "Improved a lot") separately for math, reading and writing. 
Results are displayed in Figure 6. 
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Figure 6. Self-reported ratings of improvements in basic skills. 

In each of the skill areas, the most common rating is "improved somewhat", which 42- 
48% of respondents chose to describe the extent of their skill development. Math skills 
were more frequently reported as having improved "a lot" than were reading or writing 
skills. The difference in perceived math and reading improvement is perhaps not 
surprising considering that the test score gains in math were higher than those in reading; 
furthermore, both participants and their program counselors tended to see math skills as 
needing more improvement than reading skills. 

Individuals' ratings of improvements in their reading, writing and math skills are 
moderately but not strongly correlated. Using Kendall's tau-b measure of association 
among these ordinal-scaled ratings, the following coefficients were calculated: .63 
between perceived improvements in reading and writing; .47 between reading and math; 
and .47 between writing and math. There are thus stronger associations between 
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perceived improvements in reading and writing than between reading and math or writing 
and math 26 . 

Although these results may seem consistent with the assumption that the perceived 
improvements are closely associated with assessed skill gains, the data indicate otherwise. 
There is not a close correspondence between test score gain and perceived skill 
improvement. There were statistically significant math score gains associated with each 
level of perceived math improvement (average gains of 7.06, 8.82, and 7.88 scale points, 
respectively). An analysis of variance of math gain among the three rating groups 
indicates that the differences are not statistically significant (F 2 i56 = 0.433, p = .649). 
Although it seems plausible that individuals should be able to differentiate their skill gains 
retrospectively through such reports, other research suggests that individuals generally 
have considerable difficulty reporting their literacy proficiencies on scales commensurate 
with objective assessments and standards. In the recent NALS assessment, for example, 
there was not a strong correspondence among individuals' assessed functional literacy 
proficiency and their self-reports of the adequacy of their skills for everyday purposes 
(Kirsch et al, 1993). 

Subsequent welfare utilization. Thirty-seven percent (37%) of the study participants 
received AFDC payments 27 in July of 1994, a point in time 13-15 months after they took 
the post-test and completed the questionnaire. Individuals were not necessarily receiving 
AFDC at the time of the testing (in fact, as noted above, 38% reported being employed at 
the time of the testing, and 48% had some UI earnings during the quarter in which testing 
took place). But we do know that all participants were Steps clients in the JOBS program 
at some point in the past, ranging from 1-3 years prior to testing and thus about 2-4 years 
before the subsequent AFDC status "snapshot" being examined here. This welfare 
snapshot indicator is thus a reasonable measure of the extent to which Steps clients return 
to welfare utilization at a given point in time 28 . 

Predicting subsequent welfare utilization. Linear discriminant analysis was used to 
examine the joint effects of numerous variables on future utilization of welfare by Steps. 
The dependent variable in these analyses was a binary "snapshot" indicator of welfare 
utilization during the month of July, 1994 (0=did not receive AFDC cash assistance; 
l=received AFDC cash assistance). Independent variables examined included information 
from the questionnaire and the math post test and learning gain data. The linear 
discriminant analyses assess the extent to which combinations of these independent 
variables (e.g., age, education, math test score) help predict which study participants 
subsequently received AFDC. 

26 The pattern is much the same if we Pearson correlation coefficients are used: the correlation between 
perceived improvements in reading and writing is .66, whereas that between reading and math 
improvement is .5 1 and that between writing and math is .52. 

"Individuals who left Oregon and were receiving welfare payments in other states at this time would not 
be counted as receiving welfare for purposes of this analysis. 

28 We do not have direct data about participants' AFDC status at intermediate points in time, e.g., at the 
time of post-testing. Thus these data cannot reliably distinguish long-term AFDC recipients from those 
who receive AFDC intermittently over time. 
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TABLE 4 

Discriminant Analysis of Subsequent Welfare Utilization 



Variable 

(ordered by absolute size of correlation) 


Pooled within-group 
correlation with 
canonical discriminant 
function 


F to enter or remove 


Currently working? 


.84779 


11.6076** 


Math learning gain 


.47456 


3.9784** 


Steps help you get a job? 


.16031 


1.1812 


Math post-test score 


' .14722 


0.0198 


Steps help you control your life? 


-.09682 


3.0652 


Receive GED? 


.08609 


1.6156 


Age 


.07011 


1.7983 


Years of schooling 


.06052 


0.0077 


Steps help you become self-sufficient? 


.05935 


1.0588 


Additional education/training after Steps'? 


.05131 


0.7462 


Steps help you assist your kids? 


.04661 


0.2184 


Could Steps have helped you more? 


-.04275 


0.8900 



**F to remove from canonical discriminant function is statistically significant at .05 level; 



otherwise F to enter canonical discriminant function is not statistically significant at .05 level 

Table 4 summarizes the results of this analysis. Only two of the variables shown in the 
table are statistically significant predictors of subsequent welfare utilization: math 
learning gain and currently working (i.e., whether the individual was working at the time 
of the test in the second quarter of 1993). Learning gains and employment at a given 
point in time predict less welfare utilization at a future point in time. Neither age, 
education nor any of the other variables listed in the table helped predict subsequent 
welfare utilization when currently working and math learning gain are statistically 
controlled. Interestingly, the assessed math score (math post-test score) is not a 
significant predictor, only math learning gain. Learning itself seems to have decreased 
welfare dependence among Steps clientele. 



TABLE 5 

Canonical Discriminant Function Coefficients 



Variable 


Standardized 


Unstandardized 


Currently working 


.88206 


1.8488218 


Math learning gain 


.53144 


.0584701 


(constant) 




-3.0664054 
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Table 5 displays the coefficients of these two predictive variables in the standardized and 
unstandardized versions of the canonical discriminant function used to predict future 
welfare utilization. The discriminant function correctly predicts nearly two-thirds (66%) 
of the individuals' subsequent welfare status when no prior information is utilized in the 
calculations (i.e., when being and not being on welfare at the future point in time are 
assumed to be equally likely). The 66% "hit rate" suggests that the discriminant analysis is 
moderately effective in a practical sense as well as being statistically significant. The 
function correctly identified 80% of the clients who actually were receiving AFDC at the 
time of the future "snapshot", and 58% of those who were not receiving AFDC at the time 
of the snapshot. 

Other variables not measured in this study probably would increase the predictability of 
future welfare utilization. Changing household composition (e.g., marriages, children) and 
situational circumstances are not accounted for and are known to substantially impact 
future welfare dependence (Washington State Institute for Public Policy, 1993). The 
moderate degree of predictability obtained with these two variables here is therefore of 
considerable interest. 

Predictions for the most disadvantaged clients. As noted previously, direct information 
is not available about study participants' welfare status at the time of the post-testing. All 
participants were Steps clients and thus welfare recipients at some point in the past. As a 
way of looking more closely at the relationship between learning gains and future welfare 
dependence, consider the subset of study participants who had no UI earnings at all 
during the second quarter of 1993, when the testing took place. This forms a group of 85 
out of the 163 study participants. These individuals were welfare recipients during one or 
more periods preceding the post-testing (including at a rninimum the period of their 
participation in Steps) and had no UI earnings during the entire second quarter of 1993 
when the post-testing took place. The operational assumption made here is that most of 
these past Steps clients having no UI earnings during the quarter when they tested were on 
welfare at the time of the post-testing 29 . Let us operationally term this subgroup the 
"most disadvantaged" clients. Although the group is not necessarily composed only of 
highly disadvantaged welfare recipients, many in the group are likely to be highly 
disadvantaged clients on welfare at the time of the post-testing 30 . 

The same discriminant analysis used above with the entire group of participants can be 
repeated here to predict which of these "most disadvantaged" clients are dependent on 
welfare a year later. The discriminant analysis indicates that only math learning gain is a 
significant predictor of these clients' future utilization of welfare assistance.^ 83 = 4.451, 
p =.0379). A direct comparison of the learning gains of the long-term clients who did and 
did not subsequently receive welfare is shown in Table 6. Although individuals in both 



29 A counterexample might be a participant who married after leaving Steps and needed neither cash 
assistance nor work at the time of the posl-testing to support her family. Although such cases are 
possible, they are assumed to be relatively infrequent for the purposes of this particular analysis. 
30 But they may well not be so-called "long-term recipients" of AFDC. 



9 

ERIC 



33 



44 



groups increased their math skills, those who did not draw cash assistance at the later time 
point had much larger learning gains (averaging 9.97 scale points) than those who did later 
draw gash assistance (averaging 5.54 points). An independent samples t-test indicates the 
difference between the two groups' mean learning gains is statistically significant at the 
.05 level (t = 2. 1 1, df = 83, p = .038) 31 . 



TABLE 6 

Relationship of Math Learning Gain to Subsequent Welfare Utilization 
Among "Most Disadvantaged" Clients 



Subsequent 

welfare 
utilization? 


Number of cases 


Mean math 
learning gain 


Standard error of 
mean 










No 


39 


9.97 


1.54 


Yes 


46 


5.54 


1.42 



It is should be noted that the significant relationship here appears to be between skill gain 
(learning, i.e.) and welfare rather than between skill level and welfare. The same t-test 
applied comparing the post-test scores of the two groups finds no significant difference (t 
= 0.58, df = 83, p = .562). Nor do pre-test scores differ significantly between the two 
groups (t = -1. 14, df = 83, p = .258). 



Factor analysis of data. We have looked thus far at several types of data reflecting the 
experience and perception of participants: their literacy and literacy growth (represented 
here by math test scores); their perceptions of how Steps to Success helped them 
(including improvements in their basic skills); and their employment, earnings and welfare 
utilization outcomes. We have observed significant correlational relationships among 
some of these variables while examining them in somewhat piecemeal fashion so far. In 
order to synthesize some of our findings, it may help place relationships among these many 
variables in the context of an underlying framework. 

Factor analysis is a statistical technique which can help identify a relatively small number 
of dimensions or factors underlying the relationships among many interrelated variables. 
A number of different factor analytic models were initially applied to this study's data in an 
exploratory fashion, the results of which indicated that a three-factor model is optimal for 
representing relationships among the 15 variables being considered 32 . Table 7 displays a 



3 l This is a two-tailed test based on equal variances in the two groups. A Levene test of the equality of the 
two groups' variances supports the null hypothesis of equal variances (F = 0.015, p = .904). This version 
of the t-test is equivalent to the statistical test of the F ratio in the foregoing discriminant analysis, where 
t 2 = F. 

32 A number of techniques were used to determine that 3 factors are required to adequately represent the 
variability among these 15 variables. Maximum likelihood solutions were estimated for 2, 3, 4 and 5 
factor models. Such maximum likelihood estimates, under multivariate normality assumptions, have an 
approximately chi-squared distribution from which goodness-of-fit statistics can be constructed. The 
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three-factor model from a principal components analysis. The three factors account 
respectively for 15.0, 14.0 and 1 1.2% of the variance among these variables (together, the 
three factors account for 40.2% of the variance). 

TABLE 7 

Factor Loadings for 3-Factor Principal Components Model (Varimax-Rotated)* 





Factor 1: 


Factor 2: 


Factor 3: 


Variable 


"Employment" 


"Personal 


"Educational 






Growth" 


Development" 


Currently working? 


.80931 






UI eaf lings, 2nd quarter 1993 


.78483 






Receive AFDC in July 1994? 


-:63057 






Years of schooling 








Steps help you assist your kids? 




. .70207 




Steps help you control your life? 




.69233 




Steps hehvyou become self-sufficient? 




.68203 




Steps help you get a job? 


.48660 


.51251 




Perceived improvement in math skills 




.43488 




Math post-test score 






.72707 


Additional education/training after Steps? 






.61498 


Could Steps have helped^ou more? 






.54943 


Math learning gain 






.46258 


Receive GED? 






.46205 


Age 









♦Absolute factor loadings < .35 are not shown here; Appendix A lists the complete factor pattern matrix. 

The three rightmost columns display the factor loadings of the variables. Individual factor 
loadings can range between -1 and +1; high loadings, that is, those closer to +1 or -1, 
signify stronger relationships between a variable and a factor, whereas loadings closer to 0 
indicate weaker relationships. Loadings less than .35 in absolute value are not shown in 
the table (i.e., the corresponding cells are blank) in order to focus attention here on the 
variables most influential in these factors (see Appendix A for the complete matrix of 
factor loadings). The factor matrix was rotated orthogonally into a form that minimizes 
the number of variables that have high loadings on the given factors in order to make 
factors easier to interpret (the so-called "varimax" rotation). 

The three factors, glossed in the table as "Employment", "Personal Growth" and 
"Educational Development", are readily interpretable. As the loadings in the table 



goodness-of-fit of the two-factor model has a chi-square statistic of 1 15.6 with 76 degrees of freedom, 
which is a highly significant (p = .0023) departure from the null hypothesis of adequate fit. The three- 
factor model, on the other hand, has a chi-square statistic of 69.7 with 63 degrees of freedom, for which 
the null hypothesis of adequate fit cannot be rejected (p = .262). 
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indicate, the first factor, labeled Employment, is comprised of current employment status 
at the time of the testing (2nd quarter 1993), earnings during that quarter, the perception 
that Steps had helped one to find a job, and the later (July 1994) receipt of AFDC. The 
negative sign for the factor loading of Receive AFDC in July 1994 indicates, of course, 
that not utilizing AFDC in the future is associated with increasing prior employment and 
earnings 33 . 

The second factor, labeled Personal Growth, is comprised of former clients' retrospective 
perceptions of whether Steps to Success had assisted them in key program goal areas: 
helping their kids; controlling their lives; becoming self-sufficient; getting a job; and 
improving their basic skills. The lack of association found above between clients' ratings 
of their math skill improvement and their assessed math skill gains is now easier to 
understand. The perception of skill improvement is associated with other perceived 
outcomes of Personal Growth rather than with the math scores which are part of a distinct 
Educational Development factor. It is also noteworthy that the clients' evaluation of the 
program's quality ("Could Steps have helped you more?") is part of the third (Educational 
Development) factor rather than of this Personal Growth factor. Finally, it should be 
mentioned that the "did Steps help you get a job?" variable loads heavily in both the 
Employment factor and the Personal Growth factor. 

The third factor, labeled Educational Development, is comprised of variables related to 
the development and achievement of basic skills (i.e., math learning gain, math post-test 
score) 34 , the attainment of a GED certificate (itself a mark of certain academic skills), the 
pursuit of additional (primarily vocational) education and/or training after leaving Steps, 
and the perception that Steps could have better prepared clients for the future. The 
clustering of the perception that Steps could have better prepared clients with the basic 
skills, GED and continuing education outcomes is consistent with other findings in this 
study. It was noted above that most of the open-ended responses to the question of how 
Steps could have better prepared clients (asked only if the preceding question was 
answered affirmatively) referred to additional time for skill development, continuing 
education and other activities generally seen as preparatory to job search and placement. 
Furthermore, participants in the focus group also indicated that some clients are more 
successful (at getting off of welfare and supporting their families) with an early emphasis 
on academic skill development, continuing education and vocational training before 
entering into employment. The composition of the Educational Development factor is 
certainly congruent with this interpretation. 



33 The fact that math skills and learning gains do not appear here in the Work factor is not necessarily 
inconsistent with the previous finding that math learning gain is significantly associated with decreased 
fut ;re welfare utilization. The present factor analysis looks at the relationship among a broad set of 
variables (which, in the case of the Employment factor, is dominated by very strong relationships among 
employment, earnings and welfare dependence). The earlier discriminant analyses looked in a more 
focused way at the specific relationship between a given pair of variables (i.e., learning gain and future 
welfare dependence). 

34 We could have just as well included math pre-test score here. Since any one of the three math scores 
(pre-, post-, gain) is by definition a linear combination of the other two, only two of the three need be 
included in such an analysis. 
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DISCUSSION 



Literacy Development 

Program impact on learning gains. Significant gains in clients' basic skills occurred in 
both reading and math. Gains for reading, although statistically significant, are more 
difficult than math gains to identify and associate with other outcomes in this study 
because so many of the clients scored "off the scale" on reading (i.e., were outside of the 
accurate range of the testing instruments). Most individuals whose reading skills were in 
the accurate range exhibited positive gains, which averaged about 4 CASAS scale points. 
This is comparable to the size of the reading gains reported in other CASAS pre-post 
comparisons (Fitzgerald & Young, 1994; Iowa Department of Education, 1994; Office of 
Community College Services, 1994; Sticht and Armstrong, 1993). 

The assessed math gains are considerably larger, averaging about 8 scale points, an 
increase corresponding to about 2-1/2 grade level equivalents among Oregon adults. This 
increase brought math skills up to a level higher than that of individuals with high school 
diplomas. Almost all individual learners realized these positive gains. These gains are 
larger than those generally found in careful studies of adult basic skills development, 
regardless of the assessment instrument used (Fitzgerald & Young, 1994; Iowa 
Department of Education, 1994; Sticht, 1994; Sticht & Armstrong, 1993; Office of 
Community College Services; 1994). 

As found in some other studies of adult literacy learning, there is evidence in these data 
that the magnitude of literacy learning gains is weakly associated with the amount or 
duration of basic skills instruction provided. A statistically significant, but weak positive 
correlation was found here between individual learning gains and the quantity of 
instruction provided by Steps (which varied across client cohorts). Analyzing data from 
the recently completed National Evaluation of Adult Education Programs, Fitzgerald and 
Young (1994) found evidence of a similar weak positive correlation between quantity of 
instruction and learner gains (while holding other differences among programs with 
multiple regression techniques) 35 . Sticht and Armstrong (1993), in a meta-analysis of 
relationships between the quantity of instruction and average learning gains across a 
diverse set of programs, also found weak positive correlations between the two. 

Sustained learning gains. But unlike most comparisons of pre-test and post-test scores 
in adult education research, in which both pre- and post-tests are administered during the 
course of an instructional program, the gains reported here represent sustained 
improvements in literacy skills measured one to three years after the instructional program 
ended. There is little confusion here between genuine learning and the confounding 
factors present in the customary programmatic pre-post comparisons. The learning gains 



35 Thc correlation between learning and instructional time was somewhat stronger in Fitzgerald and 
Young's analysis for an ESL as opposed to an ABE or ASE course. But ESL instruction is not involved in 
the present study. 
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observed among these Steps clients represent genuine literacy development and not the 
"practice effects" of recent test-taking, or the tendencies of having teachers to "teach to a 
test" just before a scheduled post-test, or other artifacts often present in traditional 
techniques of measuring learning gain (Venezky et al, 1993). 

Retention of learning. This study not only provides unusually strong evidence of 
sustained literacy learning, but also evidence that such gains persist over time. As Wagner 
(1994) has noted, some theorists and policy makers have argued that the learning gains 
produced by adult literacy education activities may thereafter be lessened by poor 
retention of skills learned. This study finds no evidence to support such a lack of 
retention. Examination of the relationship between assessed gain and amount of time 
since the end of basic skills instruction failed to find the significant negative correlation 
expected if learning gains progressively diminished over time. 

Impact of Steps versus other education and training activities. The cohort focus 
group, in discussing what they had learned through Steps, suggested that some individuals' 
basic skills might have improved, in part, by additional education and/or training taken 
after leaving Steps to Success. To test this possibility, the relationship between learning 
gains and education and training activities taken after leaving the program needs to be 
examined. Since details about such continuing education and training activities were 
collected on the questionnaire, the relationship can be readily examined. As noted above, 
many individuals (67%) reported taking additional education and training after leaving 
Steps. Very few, however, in detailing the courses taken, indicated that additional basic 
skills education was involved. Furthermore, a t-test of the difference in learning gains 
between those who did and did not take additional education/training found no significant 
difference between the groups (t = 1.63, df = 157, p = .168). There is thus little empirical 
support for the idea that education and training taken after leaving the program 
contributed significantly to the observed learning gains. 

Impact on GED attainment. There are also clear indications in the GED attainment 
outcomes of study participants that Steps effectively helped clients not only improve their 
literacy skills, but also to certify those skills through GED testing. As we saw, 22% of 
those entering the program without a secondary credential attained one by the time of the 
post-test. This rate is substantially higher than that of the GAIN programs studied in 
California (Martinson & Friedlander, 1994), though the entering skill levels of participants 
in the two studies was not comparable (and likely influenced the relative rates of GED 
attainment). 

There is strong evidence here that, as reported for the GAIN program, the basic education 
track within the Steps to Success program is relatively successful in assisting clients to 
obtain credentials that may help them in the labor market. Other research has shown that 
in Oregon, possession of a GED credential has salubrious effects on the employment, 
earnings and societal participation of those who did not complete high school (Reder, 
1994b). 
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Need to reconcile results of Steps and GAIN studies. Although results of the present 
study and the evaluation of GAIN'S educational outcomes (Martinson & Friedlander, 
1994) are generally in agreement on finding significant programmatic impact on GED 
attainment, the two studies differ in terms of the measured programmatic impact on 
clients' assessed literacy skills To consider these somewhat discrepant results carefully, it 
will be helpful to review some important methodological similarities and differences 
between the two studies. 

Methodological differences between Steps and GAIN studies. The two research 
efforts are similar, to begin with, in being among the very few extant studies of sustained 
adult literacy gains (i.e., where the final measurement is taken well after the end of the 
instructional program). Both studies assess sustained literacy learning gains achieved by 
welfare clientele. Each observes welfare clients' literacy abilities at a point in time well 
after their prior assignment to a mandated basic skills education component of a welfare 
reform program (the follow-up periods were roughly 2-3 years for the GAIN study, and 1- 
3 years in the Steps study). In both cases, former clients were offered a modest financial 
incentive to take a follow-up standardized literacy test and complete a questionnaire about 
their intervening employment and educational activities. 

Despite these similarities, the ways in which the two studies measured programmatic 
impact on literacy were quite different. The Steps study compared clients' literacy abilities 
using commensurate (i.e., both CASAS-scaled) assessment instruments before and after 
they participated in the instructional program.. Individual learning gains were accordingly 
measured by comparing a pre- and post-test score on the same scale. 

The GAIN study, on the other hand, could not measure an individual's learning gain, since 
the pre- and post-tests were not commensurate. A CAS AS instrument was used for the 
pre-test, but a different standardized functional literacy test, Tests of Applied Literacy 
Skills (TALS), was used for the post-test. The relationship between the two types of 
assessment instruments is not known, so no attempt was made to compare individuals pre- 
and post-test levels. Instead, the GAIN study utilized another, and methodologically 
powerful methodology to assess programmatic impacts. Clients deemed in need of basic 
education were randomly assigned to one of two service streams; one group (called the 
"experimentals") was mandated to a basic education program (GAIN), whereas the other 
(termed the "controls") entered the general county welfare program. As expected with 
such random assignment, the pre-test literacy scores and other background characteristics 
of the two groups did not differ significantly. Differences observed between the two 
groups' post-test scores are thus presumably attributable to being randomly assigned to the 
experimental GAIN program. Welfare clientele from five California counties with GAIN 
programs participated in the study. 

The TALS post-test scores of the experimental and control groups, averaging 475 for 
experimentals and 473 controls, were not statistically different 36 . When results were 

36 Scores reported here, following Martinson and Friedlander (1994), are the sum of the Document and 
Quantitative scores, each on a 0-500 point scale (summing to 0-1000). 
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analyzed by the entering CASAS skill level of the clients, there was a trend that the higher 
the entering skill level, the greater the difference between experimentals' and controls' 
TALS scores. For example, consider the scores of experimental and control clients whose 
pre-tests on reading and math were above 215 on the CASAS scale; experimentals' TALS 
post-test scores averaged 18 points higher than controls'. Conversely, among clients 
whose pre-test scores were below 215, TALS scores averaged 17 points lower than those 
of controls. Despite the apparent trend, which Martinson and Friedlander (1994) and 
Martinson (1994) take to mean GAIN-like programs tend to serve clients with higher 
incoming literacy skills more effectively, neither of these differences is statistically 
significant at the 5 percent level. 

When the TALS data were examined separately by county, there was a statistically 
significant difference (at the 5 percent level) between experimentals and controls only in 
one of the five counties involved 37 . In San Diego County, experimentals 1 scores were 
significantly higher than controls', differing by 34 scale points (out of a 1000), 
corresponding to 0.36 of a standard deviation. In the other four participating counties, 
there was no significant difference between the TALS scores of the two groups 
(Martinson & Friedlander, 1994). 

Methodological limitations in GAIN study. There is thus relatively little evidence in the 
GAIN study suggesting widespread programmatic impact on literacy development. But 
there is a problem drawing a negative conclusion from this study about the lack of broad 
programmatic impact on clients' literacy skills. As Martinson and Friedlander (1994) 
carefully point out, their statistical comparison of the experimental and control groups 
does not estimate literacy gains on a per participant basis. The experimental "treatment" 
was not participation in GAIN'S basic education program, but assignment to it. Some 
clients so assigned to GAIN (in spite of the "mandate" to attend) chose not to participate 
at all, and others participated erratically. Furthermore, some of the "controls", who were 
not assigned to GAIN, participated in other basic skills education programs. Martinson 
and Friedlander (1994) speculate that this confounding of program participants and non- 
participants in their analysis effectively reduces the size of the underlying gains made by 
actual participants, quite possibly to the extent that true underlying gains are no longer 
statistically significant. 

Reconciling the two studies. If, as Martinson and Friedlander suggest, their 
experimental methodology substantially reduces the apparent gains of those who actually 
participated in basic education, then the results of the GAIN and Steps studies may be 
easier to reconcile. The sustained learning gains observed among Steps CLP Track 
clientele do represent direct individual learning gains. Recalling the scatter plot in Figure 
3, almost all individuals increased their literacy skills, regardless of entering skill level. 
The largest program effect size on literacy growth reported for any subgroup or county in 
the GAIN study was 0.36 standard deviations, considerably smaller than the overall effect 
size of 0.77 standard deviations in the Steps study. Furthermore, there is no evidence in 

37 Although six counties were involved in the larger GAIN evaluation study, only five of these participated 
in the TALS testing. 
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the Steps study to support the conclusion some have drawn from the GAIN study, that 
only those welfare clients entering with skills above 215 on the CAS AS scale (or some 
other threshold) will benefit from a basic education program. If the program is well- 
designed and clients are appropriately screened, it appears that a basic education program 
can effectively serve welfare clients having a broad range of literacy skills. 

What program factors are associated with successful educational outcomes? 

In exploring why GAIN programs produced significant effects only in San Diego County 
(and there only for those entering with relatively high levels of skill) Martinson and 
Friedlander (1994) and Martinson (1994) suggested that San Diego County had developed 
a "unique" adult education program for GAIN clientele: 

"This result appears to be due to the unique nature of the adult education program in 
San Diego. Unlike any of the other counties, San Diego developed a new and 
separate system for providing adult education on a county-wide basis. Based on the 
premise that the existing services offered by the adult education system were not 
appropriate for the GAIN population because of their previous negative experiences 
in school, the education services were redesigned to include up-to-date computer- 
assisted instruction, off-campus locations, a new teaching staff selected for their 
strong interest and/or experience in teaching disadvantaged adults, a special class for 
learning disabled students, and close on-site monitoring." (Martinson, 1994, pp. 12- 
13) 

According to local practitioners familiar with the diverse range of adult education 
programs serving GAIN clients in San Diego County, this is not a particularly apt 
description of the diversity of programs serving GAIN clients in San Diego County (e.g., 
see McDonald, Huie, Sticht & Grimes, 1994, for a different description). Nor does the 
account tell us why only students entering with relatively high levels of skill would benefit. 
To help us better understand how program characteristics impact learner outcomes, 
research designs like that of the GAIN study may need to take the varying fit of program 
and learner characteristics into account in the design of the random assignment process. 

Unfortunately, the Steps study does not advance our knowledge in this area, either. 
Although we had originally wanted to contrast the literacy development and post-program 
life outcomes of leaeners who had and had not taken the front-loaded CLP course, we 
were not able to do so (as noted above). Even though we have much evidence here that 
the literacy education within the CLP Track appears to be unusually effective, the design 
of the study does not enable us to isolate the effects of the CLP course per se. Certainly 
the life-skills instructional focus and grouping of welfare clients in Steps literacy education 
are features shared by many (but not all) of the San Diego County programs serving 
GAIN clients. But we cannot tell from the GAIN study whether it was the students of 
those programs who experienced the skill gains while other learners did not. 
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Methodological concerns about the self-selection of participants in the Steps study. 
In considering the literacy gains broadly experienced by Steps to Success clientele 
participating in the CLP Track, it's important to examine how factors involved in the self- 
selection of study participants may have influenced the observed results. One possibility is 
that individuals in the study's target population self-selected on the basis of how they 
perceived their basic skills to have changed. For example, although all targeted former 
clients were invited by mail to participate in the follow-up study, perhaps those who felt 
their basic skills had improved were more likely to participate in the study than those who 
felt their skills had not improved. As plausible as it might seem, the results of the study 
are not consistent with such an account. There was a considerable range of perceived 
improvements in basic skills reported by study participants in the questionnaire, ranging 
from a sizable group whose members perceived their basic skills not to have changed at all 
to another group whose members perceived their skills to have improved "a lot". This 
finding in itself tends to refute the suggestion that only those former clients who felt their 
basic skills had improved chose to participate in the study, thereby biasing the distribution 
of observed gains. Since the assessed and perceived skill gains of study participants were 
found not to be statistically related,, it seems quite unlikely that self-selection by members 
of the target population on the basis of their perceived gains might have appreciably biased 
the overall distribution of assessed gains. 

Although it was noted previously that clients who accepted the invitation to participate in 
the study had significantly higher pre-test scores than those who did not accept the 
invitation, there is little reason to suppose that this materially biased the gain observed 
among those who did participate. Recalling the scatter plot of pre-test and post-test 
scores examined in Figure 3, it is clear that the vast majority of clients over a broad range 
of pre-test scores individually experienced positive gains. The most likely explanation for 
the observed difference between the pre-test scores of the study participants and non- 
participants is that there is a negative correlation between individuals' literacy skills and 
their degree of "test-averseness". Individuals who tend not to score well on standardized 
basic skills tests, according to this account, would be less likely to take another test. 
Although this hypothetical process would bias the distribution of pre-test scores of those 
who do take the test, it should not in itself bias the distribution of their observed gains. 
Stronger assumptions would be needed to materially bias the distribution of self-selected 
test-takers' gains. Although such hypothetical possibilities cannot be entirely ruled out 
without data from an appropriate control group (and we saw above in the GAIN study 
how hard it can be to construct appropriate control groups), there is no evidence that such 
hypothetical factors are operating in the Steps study. 

Perhaps other variables, such as individuals' employment and welfare status at the time 
study participants were being recruited, influenced the self-selection process in ways that 
biased the distribution of observed skill gains. Although data on such variables is not 
available for non-participating members of the target population, there is little reason to 
suppose that the possibility of self-selection based on these variables should limit the 
generality of the conclusions we draw from the analysis of those who did participate. The 
study participants encompassed a broad range of economic circumstances at the time of 
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the testing, as reflected in their employment and earnings data (whether taken from self- 
reported or from "hard" sources). The preponderance of individual participants 
experienced learning gains, regardless of whether they were working full-time, part-time 
or not at all during that period of time (and, as argued above, many in the last group were 
likely to have been on welfare at the time). Although it is certainly conceivable that the 
population estimates of learning gains among the target clientele are biased by some 
degree of non-representativeness in the self-selected study sample, there is little evidence 
to suggest that the widespread literacy gains that were observed might somehow have 
vanished if the study participants had been sampled at random From characteristics we 
were able to compare across study participants and non-participants, it is clear that 
participants were relatively but not fully representative of the target population. Although 
study participants are not a statistically random sample of the target clientele, it can be 
said with a reasonable degree of confidence that the program effectively increased the 
literacy proficiency of participants encompassing a broad range of economic 
circumstances, educational backgrounds and initial literacy abilities. 

Several other possible sources of bias considered, as well. Many opinions were offered — 
by the cohort focus group, by the testing administrators and other Steps staff, by other 
JOBS program and welfare officials — as to who from among those invited to participate 
in the study ~ might selectively volunteer to do so. The range of suggestions offered 
reflects a diverse set of outlooks on and understandings of the program, its clientele and 
the characteristics of "the system". For example, some staff predicted that "only those 
needing the money" would turn out, whereas another felt that former clients would come 
"out of loyalty to the program". A third suggestion was that "only people doing 
something" (i.e., working or going to school) would turn up. None of these suggestions 
turns out to accurately characterize the full range of study participants. As shown above, 
the participants cut across a broad range of employment and income situations at the time 
of the testing. A significant number were "doing nothing" at the time: they had neither 
taken additional education or training since leaving Steps nor were employed at the time of 
the testing. 

In sum, although it is clear that the realized sample of study participants is not fully 
representative of the target population in a statistical sense, there is little reason to 
suppose that study participants self-selected on a narrow basis or one that seriously limits 
our ability to generalize the learning gain outcomes to the target population of clientele. 



Impact of Literacy Development on Welfare Dependence 

A primary rationale for including basic skills education in JOBS programs such as Steps to 
Success, of course, is to facilitate clients' long-term independence from the welfare system. 
The assumption has been, as discussed in the introduction, that lack of educational 
credentials and poorly developed literacy skills are barriers to attaining and maintaining the 
employment and earnings needed to support a family. 



Some "hard" evidence of economic impact. This study provides clear evidence that not 
only can appropriate basic education services increase welfare clients' literacy skills, but 
that these learning gains are significantly linked to future reductions in welfare 
dependence. The evidence for this are statistical tests of whether clients' literacy learning 
gains (measured during the second quarter of 1993) were associated with their subsequent 
welfare status in a "snapshot" taken during July, 1994. Statistical tests conducted 
indicated that the larger the assessed learning gain, the less likely a client was to be on 
welfare in the future. Furthermore, increased literacy appears to be particularly potent as 
a predictor of future self-sufficiency among clients operationally defined here as being the 
most disadvantaged (i.e., those who had no earnings at all during the second quarter of 
1993). These results may be the first hard evidence that programmatically stimulated 
literacy development reduces clients' future welfare dependence. 

The role of literacy education in welfare-to-work programs. The discriminant analysis 
which predicted clients' future welfare dependence from other variables measured in the 
study found two variables to be important: literacy growth and current employment. This 
result suggests that both kinds of activity tracks in Steps (i.e., the CLP and Placement 
Tracks) may be important to maintain within future welfare reform programs. This 
finding, coupled with the finding that literacy growth alone (and not current employment) 
predicts future welfare dependence among the "most disadvantaged" clients suggests a 
human capital approach to welfare reform should be utilized in some contexts. 

Cautionary notes. Comparable welfare status data are not available for target Steps 
clients who did not participate in the study. But there is no particular reason to suppose 
that the self-selection process operating during study recruitment is responsible for the 
relationship observed between study participants' learning gains and their future welfare 
dependence. Recall that we were relatively confident in generalizing learning gain 
outcomes from study participants to non-participants in part because of the comparisons 
we were able to make between the educational and literacy characteristics of the two 
groups. It would be prudent, however, to be more cautious in generalizing results about 
future welfare dependence because there are not comparable data available about the 
employment, earnings and welfare status of the two groups. Nevertheless, it can be 
confidently stated say that the results relating literacy growth to future self-sufficiency 
seem to hold for clients from a broad spectrum of economic circumstances. 

It is important to point out, however, that these results do not constitute direct evidence 
that participation in Steps CLP Track decreases clients' future dependency rates. Direct 
evidence for that would require finding an overall difference in dependency rates between 
comparable client populations participating and not participating in the CLP track. All 
that the present results indicate is that among those who do participate in the CLP Track, 
the larger the literacy gains they achieve, the less likely is their future welfare dependence. 
And this holds especially for the apparently "most disadvantaged" clients. 

Needed research. There are two ways in which the needed overall comparison might be 
obtained. One technique would involve gathering information about a sizable population 
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of program clients who met Steps eligibility criteria for the CLP Track. Information 
would be culled from agency records about client background characteristics, participation 
in various program activities (including CLP and basic skills education, and post-Steps 
employment and welfare outcomes. Assuming that not all Steps clients eligible for the 
CLP Track were in fact assigned to it (a tenable assumption because of the limited number 
of "slots" available in that track), outcomes of those participating and not participating in 
the CLP Track could be compared, either matching groups or controlling for group 
differences statistically. Such an approach might have been feasible in the present study 
had outcome data been available for appropriate groups of clients 38 . 

An even stronger approach might be the experimental or random assignment methodology 
used by Manpower Demonstration Research Corporation (MDRC) in its eval ition 
studies of the GAIN program 39 . This would entail randomly assigning some clients 
deemed in need basic education to the CLP track and others to the Placement Track. As 
overall GAIN evaluation studies (Friedlander, Riccio & Freedman, 1993) illustrate, this 
experimental method pioneered by MDRC allows differences in measured outcomes 
between experimentals and controls to be unambiguously attributed to the experimental 
program. Although this approach seems to work well for evaluating overall program 
impact, as we saw in discussing the evaluation of GAINs educational outcomes, it can be 
difficult to isolate the impact of the basic educational components within a welfare-to- 
work program. Further methodological refinements are needed. 

Client Perceptions of Literacy Development 

Members of the cohort focus group, the testing session administrators (many of whom had 
extensive experience with other Steps clientele), and numerous program staff and JOBS 
program officials perceived that some students improved their basic skills (by going 
through the CLP Track) while others did not. That is exactly what the. study participants 
reported about whether their own skills had improved: Some of the questionnaire 
respondents reported that their skills had improved, whereas others did not. That much 
seems straightforward and unremarkable. 

What is striking, however, is the lack of relationship found between individuals' reports of 
improvements in their basic skills and assessed gains in those skills. As surprising as such 
results may seem, they are consistent with other research which shows how poorly adults 
evaluate the adequacy or growth of their literacy skills in relation to external criteria. In 
the recently completed NALS study, for example, the vast majority of individuals whose 
skills were judged to be lacking in terms of actual performance nevertheless felt their skills 
were adequate for everyday purposes (Kirsch et al, 1993). 



38 New regulations were put into place during the course of the study which severely limited access to 
client records. 

i9 Steps to Success is in fact now participating in an MDRC evaluation study of Oregon's JOBS program; 
unfortunately, educational outcomes arc not included in the study. 
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At the same time, it should be noted that some skill differences are very apparent to adult 
learners. Members of the cohort focus group and many of the clients participating in the 
testing (according to their test administrators) remarked about how much better their 
reading skills were than their math skills. And how much "harder" the math test seemed 
than the reading test. These perceptions correspond to an overall 20 scale-point difference 
between the average reading and math scores, suggesting perhaps that large enough skill 
differences are accurately self-perceived. Perhaps the smaller math gains clients 
experienced, which may have accrued slowly overtime, were just not large enough to be 
reliably noticed. 

Members of the cohort focus group, in discussing their just completed experience of 
taking the test and completing the questionnaire, emphasized how poorly they felt that 
"that kind of test measures the learning almost everyone felt did occur in Steps. These 
learners felt that the reading end math tests did not measure what they had learned in 
Steps, which they said emphasized "personal growth", "endurance", "how to stay off 
welfare", "how to advocate for yourself' and other aspects of the "whole person" and 
"goal-related" accomplishments. From one perspective, it sounds as if the focus group 
was describing the learning they most valued or which had helped them the most, which 
was not the basic skills instruction but the content of the Career and Life Planning course. 
But when probed to elaborate, it became clear that they felt the improvements which some 
had made in reading, writing and math could not be measured separately from the other 
learning that had taken place at the same time. 

There were also indications from many of the Steps staff that they felt standardized tests 
like BASIS and CASAS were "not aligned" with the content of what they were teaching 
and what clients were learning. There again seems to be a perception that what was 
taught (and assessed) "on the other side of the building" (the ABE/GED classrooms, i.e.) 
was not appropriately integrated with the broader and more personally-relevant material 
taught in Career and Life Planning. 

It is not entirely clear what role such divisions of content might play in the apparent non- 
alignment between clients' self-perceived and assessed gains in basic skills. But, whatever 
its genesis, the lack of correspondence between perceived and assessed learning represents 
something significant to clients. It should be considered as much more than a mere 
methodological or theoretical curiosity. When adults' perceptions of the adequacy of their 
skills do not correspond with those of program personnel and policy (which are often test- 
driven), it may be extremely difficult to motivate their participation in basic skills training 
(Reder & Green, 1985; Wikelund et al, 1992). Program retention may further deteriorate 
if learners' perceptions of their skill development do not correspond with indicators of 
progress used by program policy and personnel. 
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Three Factors in Search of a Pronram: Employment, Personal Growth and 
Educational Development 



When the many variables involved in this study were represented in terms of a few 
underlying constructs using the techniques of factor analysis, an interesting pattern 
emerged which can inform the foregoing discussion about client perceptions of literacy 
learning and program impact. Recall that the factor analysis identified three major factors 
underlying client outcomes and perceptions, labeled Employment, Personal Growth and 
Educational Development. The Employment factor — on which load the outcomes of 
employment, income, welfare dependence and the perception that Steps helped clients get 
a job — seems straightforward to understand. It corresponds to clients' experiences of 
leaving Steps and entering the world of work and self-sufficiency for themselves and their 
families. This, of course, is the primary policy and programmatic goal of Steps to Success 
and other welfare-to-work programs. 

The other two major factors, Personal Growth and Educational Development, correspond 
to the main strategies for assisting CLP Track clients to accomplish the goal of 
Employment, the welfare-to-work transition. Former clients' perceptions about whether 
Steps to Success had assisted them in key areas — helping their kids; controlling their 
lives; becoming self-sufficient; and getting a job ~ loaded highly on Personal Growth, 
along with perceptions of how much their basic skills had improved. The fact that 
perceived improvements in literacy skills are part of Personal Growth rather than 
Educational Development (where assessed gains in basic skills load) may be very 
important. If adult literacy learners' perceptions of growth in their basic skills influence 
their participation in instruction, then these results indicate that participation in adult 
education may depend on broader perceptions that learners' felt needs for Personal 
Growth are being met. This pattern of perceptions among Steps literacy students is 
certainly consistent with emerging new theories of participation for adult learners (e.g., 
Wikeland et al, 1992). 

The variables of literacy development, GED attainment, taking additional education or 
training after leaving Steps, and the perception that Steps could have better prepared 
clients all load highly on the Educational Development factor. The fact that the 
perception that Steps could have better prepared clients loads on Educational 
Development (rather than on Personal Growth where other perceived impacts of Steps 
load) together with outcomes for assessed basic skills improvement, GED attainment and 
continuing education is consistent with other findings in this study. Most open-ended 
responses to the question of how Steps could have better prepared clients referred to 
additional time for skill development, continuing education and other activities generally 
seen as preparatory to job search and placement. Furthermore, the cohort focus group 
suggested that some clients are more successful (at getting off of welfare and supporting 
their families) with an early emphasis on academic skill development, continuing education 
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and vocational training before entering into employment. Other clients may not be so 
favorably disposed towards or helped by Educational Development. 40 

The factor analysis of client perceptions and outcomes is consistent with the rich and often 
heartfelt comments heard from the cohort focus group and throughout Wikelund's (1993) 
in-depth qualitative research. Most welfare clients reported experiencing a distinction if 
not a tension between "personal" and "academic" growth. Frequently, the two types of 
growth were perceived as being in opposition to one another within the priorities of Steps 
and within the larger welfare system clients negotiated. Although it may seem reasonable 
from a programmatic perspective to separate basic educational services into two 
components (i.e., into Career and Life Planning, which directly addresses Personal 
Growth, and ABE/GED which relates directly to Educational Development), many clients 
expressed strong opinions about not wanting to lose more CLP-like support as a price of 
receiving basic skills instruction. Most clients valued most the "personal" growth that had 
taken place at Steps. Among clients who had attained economic self-sufficiency felt in . 
retrospect that the personal growth stimulated by Steps was crucial in successfully making 
a transition from welfare-to-work: "We learned how to make a commitment"; "we got 
endurance and self-reliance"; "we learned how to communicate with people"; "how to 
advocate for ourselves"; "how to stay off welfare"; "how to have confidence in myself. 

Many clients wanted the kinds of support they received from CLP to have continued after 
the end of the CLP course, while participating in basic skills instruction, and even after 
finding a job, leaving Steps and starting to work. "That's the time," one client remarked 
referring to the difficulties of making a transition to full-time employment, "when you 
really need support from others dealing with the same problems — trying to take care of 
your kids and work at the same time!" They suggested that even continuing some of the 
support-group and counseling activities, launched during CLP, on perhaps a monthly basis 
during the evenings after work would be extremely helpful to those struggling to "make it" 
in the workplace 41 . 



Discussions with Steps staff also indicated that they, too, believe that finding ways to 
integrate educational, counseling and group-support services on a longer-lasting, post- 
employment basis is a key to having greater programmatic impact. Staff and cohort 
members alike feel that literacy education will be more effective in the long run if it is 
integrated with other instructional and support activities (such as those offered in CLP). 
One staff member noted that "providing basic skills instruction based on a holistic view of 
clients encourages personally relevant participation and skill development". 

There are thus some indications here that providing services in a more integrated or at 
least in a temporally overlapping manner might better meet the needs of clients and assist 



40 Interestingly, Educational Development is the only factor of the three whose factor scores have a 
bimodal distribution among clients. Whether this corresponds to there being two underlying "types" of 
clients in terms of orientation to Educational Development requires further analysis. 
41 A related development is now taking place at Steps. A grant was received specifically to provide 
services in this fashion after clients begin working. 
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them to reach their goals in the areas of Employment, Personal Growth and Educational 
Development. One key to stimulating even more effective basic skills instruction (and 
there's ample evidence in this study that the existing programs have already been quite 
effective) may be to imbed and intertwine literacy instruction with program components 
such as CLP, rather than structuring them as separate activities to be taken in sequence. 

These results should challenge policy-makers and program designers to re-examine their 
assumptions about the ways in which "hard" outcomes (e.g. welfare dependence, 
employment, earnings, assessed literacy skills) and "soft" outcomes (e.g., self-esteem, felt 
needs, perceived improvements of basic skills) are related to each other and to literacy 
development among welfare clientele. The findings of the study suggest that too often 
they are assumed to be independent of each other. Holistic views of both clients as 
learners and of programs as learning environments may be a useful starting point for this 
re-examination. 
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APPENDIX B 
Sample Correspondence and Administration Procedures 



Northwest 
Regional 
Educational 
Laboratory < 




101 S.W. ivlain Street, Suite 500 • Portland, Oregon 97204-3297 
(503) 275-9500 GTE: NWREL.LAB FAX: (503) 275-9489 



Writer's Direct Dial Number 



March 17, 1993 



Dear Steps To Success graduate: 

You have the opportunity to receive $25 for participating in a brief activity. We are 
conducting a research project looking at the impact of programs like Steps To Success 
on adults' lives - do they make a difference? We hope our findings can help these 
programs to improve. 

We are inviting you - and a sample of others who went through Steps ~ to come to a 
brief meeting to provide us with some information for our study. At the meeting, you 
will fill out a brief questionnaire and take a multiple choice skills test (similar to one 
you probably took when you entered Steps). 

The information you provide will be confidential. No individual information or test 
scores will be given to any agency. 

If you participate in one of these sessions (lasting about 1-1/2 hours) you will receive a 
$25 stipend at that time. Because the stipend is intended to cover your costs of 
participating, it will not be reported to the Internal Revenue Service or Oregon 
Department of Revenue. If you are currently receiving a welfare grant, in most cases it 
will not count as part of your earnings disregard. (Please see the attached letter from 
ML Hood Community College or check with your caseworker if you have a question 
about this.) 

You can come to one of the sessions listed on the back. We hope one of the dates, 
times and locations will be convenient for you. Parking is available at all locations. 

We hope you will be able to participate. So we can have your check ready, you must 
pre-register for a session by returning the enclosed postage-paid reply card. Space is 
limited. Please return the card as soon as possible. After receiving your reply card, 
we will send you a confirmation notice. 

We look forward to seeing you. Thank you for your assistance. 



Sincerely, 



Stephen Reder 
Program Director 
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AN EQUAL OPPORTUNITY EMPLOYER 



Test Session Dates, Times and Locations 

Portland Community College 
Southeast Center 
2850 S.E. 82nd 
Portland, Oregon 97266 

Session #1: Monday, April 5, 1993, 7:00 p.m., Room B-16 
Session #2: Wednesday, April 7, 1993, 2:00 p.m., Room B-22 
Session #3: Tuesday, April 13, 1993, 3:30 p.m., Room B-16 
Session #4: Saturday, April 17, 1993, 10:00 a.m., Room C-4 
Session #5: Thursday, April 22, 1993, 7:00 p.m., Room C-2 
Session #6: Saturday, April 24, 1993, 1:00 p.m., Room D-l 

Mt. Hood Community College 
Centennial Mt. Hood Center 
14750 S.E. Clinton Street 
Portland, Oregon 97236 

All tests will be given in Room 2. 

Session #7: Wednesday, April 7, 1993, 7:00 p.m. 

Session #8: Tuesday, April 13, 1993, 7:00 p.m. 

Session #9: Friday, April 16, 1993, 10:30 a.m. 

Session #10: Thursday, April 29, 1993, 1:30 p.m. 

Session #11: Saturday May 1, 1993, 9:00 a.m. 

Session #12: Saturday May 8, 1993, 9:00 a.m. 

Mt. Hood Community College - Main Campus 
26000 S.E. Stark Street 
Gresham, Oregon 97030 

All tests will be given in Room 1008 (lower level). 

Session #13: Saturday, April 10, 1993, 9:00 a.m. 

Session #14: Wednesday, April 14, 1993, 1:00 p.m. 

Session #15: Tuesday, April 20, 1993, 7:00 p.m. 

Session #16: Wednesday, April 21, 1993, 2:00 p.m. 

Session #17: Saturday, April 24, 1993, 10:00 a.m. 

Session #18: Tuesday, April 27, 1993, 7:00 p.m. 
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Northwest 
Regional 
Educational 
Laboratory < 




101 S.W. Main Street, Suite 500 • Portland, Oregon 97204-3297 
(503) 275-9500 GTE: NWRELLAB FAX: (503) 275-9489 



Writer's Direct Dial Number 



May 24, 1993 



Dear Steps To Success graduate: 

This is it! "For a limited time only",.. You have the opportunity to receive $25! If you 
couldn't participate in the Steps to Success research project in April, you now have another 
chance! 

As you may recall, we are conducting a study to look at the impact of programs like Steps 
to Success on adults' lives — do they make a difference? We hope our findings can help 
these programs to improve. 

We are inviting you — and a sample of others who went through Steps — to come to a 
brief meeting to help us with our study. At the meeting, you will be asked to fill out a 
very short questionnaire and take a multiple choice skills test (similar to one you probably 
took when you entered Steps). 

The information you provide will be confidential. No individual information or test scores 
will be given to any agency. 

If you participate in one of these sessions (lasting about 2 hours), you will receive a $25 
stipend at that time. Because the stipend is intended to cover your costs of participating, 
it will not be reported to the Internal Revenue Service or Oregon Department of Revenue. 
If you are currently receiving a welfare grant, in most cases it will not count as part of 
your earnings disregard. (If you have a question about this, please check with your 
caseworker.) 

You can choose one of the sessions listed on the back of this letter. We hope one of the 
dates, times and locations will be convenient for you. Parking is available at all locations. 

We need your help now and hope you can come! So we can have your check ready for 
you, you must pre-register for a session by returning the enclosed postage-paid reply 
card. Please return the card as soon as possible. After receiving your reply card, we will 
send you a confirmation notice. 

We look forward to seeing you. Thanks for your help! 



Sincerely, 




Stephen Reder 
Program Director 



Enclosure 
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AN EQUAL OPPORTUNITY EMPLOYER 



Test Session Dates, Times and Locations 



Portland Community College 

Southeast Center 
2850 S.E. 82nd 
Portland, Oregon 97266 

Session #21 : Saturday, June 12, 1993, 10:00 a.m., Room C-2 
Session #22: Tuesday, June 15, 1993, 2:30 p.m., Room C-2 
Session #23 : Thursday, June 1 7, 1 993, 6:30 p.m., Room C-2 



ML Hood Community College 

Centennial Mt. Hood Center 
14750 S.E. Clinton Street 
Portland, Oregon 97236 

Session #24: Monday, June 14, 1993, 6:00 p.m., Room 2 
Session #25: Wednesday, June 16, 1993, 2:30 p:m., Room 2 
Session #26: Saturday June 19, 1993, 10:00 a.m., Room 2 
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INSTRUCTIONS FOR TEST ADMINISTRATORS 



1. Please show up at the test location 15 minutes before the scheduled time to allow time for getting set 
up. 

2. You should bring to each session : 

o 25 CASAS Reading and 25 CASAS Math test booklets 

o printed roster ofpreregistered participants for the particular session 

o 50 scanable answer sheets, precoded with session number and other info (25 Reading , 25 Math) 
o 40 No. 2 pencils and 25 pens 
o 25 questionnaires 

o batch of preprinted Claim for Stipend forms for the given session (number varies with roster length) 
o printed NWREL checks for participants (number varies with roster length) 
o Ink pens for signatures 
o A few pencil sharpeners 

3. If an emergency situation arises in which you cannot arrive at the testing location on time, you MUST 
contact the backup person who will work with you to figure out how to get the materials from you and get 
to the testing location on time! 

4. Before participants arrive, place a pen and a No. 2 pencil on each individual desk. 

5. As participants arrive, check each one off against your roster ofpreregistered participants. 

6. Only individuals on the roster for a given session may participate in that session. If individuals 
turn up who are not on the roster for the given session, they may not under any circumstances participate. 
They may contact Sherryl Glenn at NWREL (M-F 8:30-5:00) at 275-9592 to arrange a futur* testing 
session (assuming they previously received an invitation from us). 

7. As soon as all preregistered participants have arrived or 7 minutes have past since the designated 
starting time (whichever comes first), close the door to the room. Individuals who arrive after you close 
the door (for whatever reason) cannot participate. They may call Sherryl Glenn at NWREL (M-F 
8:30-5:00) at 275-9592 to schedule another testing session. 

8. Give each person a questionnaire and Claim for Stipend form. Ask them to complete the questionnaire 
and sign the Claim for Stipend form in ink and put their Social Security Number on it (This should take 
about 5-7 minutes.) Collect the completed questionnaires and Claim forms and the pens, making sure 
that both the questionnaire and the form have the individual's name and Social Security Number on them. 
(Some items on the questionnaire are voluntary and may be left blank.) 

9. Distribute the Reading test booklets with Reading answer sheets inserted. Have participants fill in 
their names and social security numbers carefully in the appropriate sections. Read the instructions aloud 
for using the answer sheets. Have participants answer the practice items on the answer sheet Explain the 
way to mark "X" over an item they want to erase, saving them to erase completely for the end of the 
session. 

10. Read the instructions aloud for the Reading questions. Remind them not to mark the test booklet in 
any way. Tell them there will be 40 minutes available for the items, and that if they finish early, they 
must remain quietly seated untii everyone has finished or the 40 minutes have expired. Write the ending 
time on the board. 



1 1 . Begin the Reading session. Allow 40 minutes maximum. Be sure to monitor the session and make 
extra No. 2 pencils available as needed. 

12. While participants are taking the Reading test, make sure that the information on the roster and on 
the questionnaires match. Check each Claim for Stipend form again to see that both signature and Social 
Security Number have been filled in appropriately. If the form is correctly completed, tear out the yellow 
copy and place it with the individual's check to be handed out at the end of the testing session. 

13. As soon as everyone has finished or 40 minutes have expired, ask participants to close their test 
booklets and take one minute to erase their "X"-ed out answers. Collect the answer sheets and test 
booklets. Make sure each answer sheet has a name and correct social security number on it 

14. Negotiate a firm break period of 5-10 minutes. Make it clear that the shorter the break, the sooner 
they'll be finished. 

15. Once the break is over, close the door to the room and distribute the Math booklets, with scratch 
paper and Math answer sheets inserted. Have participants fill in their names and social security numbers 
on the answer sheets. Have them fill in the practice items on the answer sheets. 

16. Read the instructions aloud for the Math questions. Remind them not to mark the test booklet in any 
way - that's what the scratch paper is for! Tell them there will be 40 minutes available for the items, and 
that if they finish early, they must remain quietly seated until everyone has finished or the 40 minutes 
have expired. Write the ending time on the board. 

17. Begin the Math section. Allow 40 minutes maximum. Be sure to monitor the session and make extra 
No. 2 pencils available as needed. 

18. As soon as everyone has finished or 40 minutes have expired, ask participants to close their test 
booklets and take one minute to eras 2 their "X"-ed out answers. 

19. When everyone has finished cleaning up their answer sheets, ask participants to form a line to bring 
their test booklets, answer sheets, scratch paper and pencils to you so they can receive a copy of their 
Claim for Stipend and their stipend check. 

20. Collect a test booklet, answer sheet, scratch paper and pencil from each person. Make sure that each 
person's name and SSN are appropriately recorded on the answer sheet. 

21. To receive a check, each individual must have completed and turned in a questionnaire, a Claim for 
Stipend form, and two answer sheets (one Reading and one Math) - all of which you have checked 
against the roster. If all of these are complete, give them their check and the yellow copy of the Claim for 
Stipend form, and thank them for their participation in this project 

22. At the end of the session, be sure the roster checklist is complete and reorganize the test booklets, 
pens, pencils and other materials you'll need for the next session. . Also, please be sure that left over 
checks are returned to NWREL along with the completed answer sheets, Claim for Stipend forms and 
questionnaires. 
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APPENDIX D 
QUESTIONNAIRE 



Name Social Security # - - 

Age Highest school grade completed 

About when did you enter the Steps to Success Program? Month Year 

About when did you leave the Steps to Success Program? Month Year 

Did you go to a Life Skills/Career and Life Planning course in Steps? Yes No 

If "yes", about how often did you go? hours a day days a week for weeks 

About how often did you attend ABE/GED classes at Steps? 
hours a day, days a week for weeks 

How much do you feel your skills improved? 

Reading No change Improved somewhat Improved a lot 

Math No change Improved somewhat Improved a lot 

Writing No change Improved somewhat Improved a lot 

Since you completed the Steps program, have you participated in any additional education 
or training? Yes No If "Yes", list each education or training course you took: 

Course Name How many months? 



In which areas has Steps to Success 
Getting a job : Yes _ 

Helping my kids: Yes _ 

Controlling my life: Yes _ 

Improved self-sufficiency: Yes _ 

As you think about your life since you left the program, do you feel there is some way that 

Steps to Success could have better prepared you ? Yes No 

If "Yes", please describe: 



Are you employed at the moment? Yes No 

If "Yes", what kind of work? Is it: Full-time Part-time 



made a difference to you? 

No_ 

No_ 
_ No_ 

No 
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* APPENDIX C 

Additional 'Information about BASIS Test 

BASIS Score Interpretation 



Score: < 200 
Level A 



Difficulty with basic literacy and computation skiills necessary 
to function in both employment and in the community 
Difficulty providing basic personal identification in written form 
(job applications) 

Unable to compute wages and paycheck deductions 
Unable to follow basic simple written directions 



Score: 200-215 
Level B 



Score: 216-230 
Level C 



Score: 231-235 
Level C 
(Pre-GED) 



Score: 236+ 
GED 



Possessing low literacy skills 

Difficulty pursuing other than entry level programs requiring 
minimal skills . 
Can fill out simple forms and perform basic computations 
Functioning below 5th grc.de level 

Able to handle basic literacy tasks and basic computational 
skills in a functional setting related to employment 
Capable of minimum wage job if carefully selected 
Difficulty following more complex sets of directions 

Can usually perform work that involves following oral and 
written directions in familiar and some unfamiliar situations 
Functioning at a junior high level in basic reading and math 
Could benefit from preliminary GED preparation 

• Functioning at a secondary level in reading and math 

• High probability of passing the GED with some preparation 
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BEST COPY AVAILA, 



HAMILTON LOCAL SCHOOLS 
Columbus, Ohio 43207 



ADDiication For Use of Personal Leave 



Employee ' s flarae 



Date(s) For Which Personal Leave is Requested 



Purpose for Which Leave is Requested 



S iynaturc 
Date 



.>c?r::vec Not /ooroved 



Principal 



Conn lets this fern in triplicate. One copy will be returned after 
acted on by the Principal; one copy retained by the Principal- one 
copy to the Superintendent's Office. 
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HAMILTON LOCAL SCHOOLS 
Columbus, Ohio 43207 
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ADoiication For Use cf Personal Leave 



Employee ' s Maine 



Date(s) For Which Personal Leave is Requested 



Purpose for Which Leave is Requested 



Signature 
Date 



iC proved r.'ot Acoroved 



Prmciaai 



Complete this form in triplicate. One copy will be returned afzer 
acted on by the Principal; one copy retained by the Principal; one? 
copy to the Superintendent's Office. 
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ADDlication For Use cf Personal Leave 



Employee ' s Marae 



Date(s) For Which Personal Leave is Requested 



Purpose for Which Leave is Requested 



Signature 
Date 



.»c?rovec Mot A-rorovod 



Princisu. 



Complete this tern in triplicate. One copy will be returned after 
actea on by the Principal; one copy retained by the Pt i ncii.n 1 ; 
copy to the Superintendent's Office. 



77 



